The same question can produce different wording and levels of detail. To make a small example inspectable, we sent six fixed Japanese tasks to three provider APIs on October 30, 2026 and published the complete inputs and responses.
This is a small illustrative provider-API study. It is not a Chappie in-app benchmark or an evaluation of the consumer ChatGPT, Claude or Gemini products. The models were available small/general models, not matched price or capability tiers. We ran each task/model pair once. These results do not establish a Japanese-language quality ranking or an overall winner.
Chappie’s developer publishes this article and dataset. The tasks are synthetic examples, not customer conversations, and contain no customer personal data.
We recorded the checks before generation
The criteria were registered at October 30, 2026, 22:20:29 UTC, before the responses were generated. Each API received the exact Japanese prompt below, without a system prompt, web search or external tools. Sampling used provider defaults, with a maximum output of 2,048 tokens. OpenAI and Google used their supported low-thinking settings; no additional Anthropic model settings were supplied.
| Provider | Requested model | Setting notes |
|---|---|---|
| OpenAI | gpt-5.4-mini |
Low reasoning, storage disabled; returned model ID gpt-5.4-mini-2026-03-17 |
gemini-3.8-flash |
Low thinking | |
| Anthropic | claude-haiku-5-5 |
No additional model settings |
All 18 responses finished with reason stop. This records completion, not a perfect instruction-following score.
What we observed
| Task | Observation in these outputs |
|---|---|
| Arithmetic | All three returned a total of 780 yen, discount of 78 and payment of 702; currency-unit and separator formatting differed |
| JSON extraction | All three returned JSON without code fences, with the requested three keys and Tokyo, 14:30 and boolean true values |
| Contradictory notes | All three separated agreement on Tokyo from the November 11/12 conflict and did not choose a date |
| Summary | All preserved the November 10 launch, Tanaka owning copy, and the unresolved image owner and budget; bullets and emphasis differed |
| All used two sentences, retained the date/time and did not invent a reason; politeness and length differed | |
| Constrained schedule | All kept three items and 10/20/30 minutes; Google used minimal labels while OpenAI and Anthropic elaborated the activities |
For a strict instruction to provide only activity and duration, the more elaborate schedule answers may require editing. Readability and adherence are separate judgments. We did not use independent blinded raters or calculate an overall score.
The exact prompts
These are the inputs actually sent, left in Japanese so they can be rerun without a translation changing the task. The downloadable JSON contains the preregistered criteria, full responses, timestamps, usage and finish reasons for every model/task pair.
Japanese email rewrite
次のメモを取引先への丁寧なメール本文にしてください。本文は2文だけにし、件名・挨拶の定型文・署名は付けないでください。日付や時刻を変更せず、理由を追加しないでください。メモ：11月6日の打ち合わせを15時から16時に変更したい。都合を確認してほしい。Summarize supplied notes
以下のメモだけを使い、決定事項・担当者・未決事項の3行に要約してください。不明な情報は不明と書き、推測しないでください。メモ：新しい案内ページを11月10日に公開する。原稿は田中が担当。画像の担当者はまだ決まっていない。会議で予算の金額は決めなかった。Check arithmetic
ノートを1冊180円で3冊、ペンを1本120円で2本買います。商品合計に10%の割引があり、ほかの費用はありません。商品合計、割引額、支払額の順に、項目名と整数の金額だけを3行で答えてください。Extract structured data
次の文からJSONだけを出力してください。コードブロックや説明は不要です。キーはcity、time、confirmedの3つだけにしてください。cityとtimeは文字列、confirmedは真偽値にしてください。文：東京で14:30に集合。参加は確定しています。Plan with explicit constraints
読書会の進め方を提案してください。合計60分で、導入10分、個人で読む時間20分、意見交換30分にしてください。箇条書きは3項目だけ。各項目には内容と分数だけを書き、追加の活動や前置きは入れないでください。Notice contradictory evidence
次の2つの資料だけを使って回答してください。資料A：説明会は11月11日、会場は東京。資料B：説明会は11月12日、会場は東京。質問：説明会の日付と会場を確定できますか？一致する情報と食い違う情報を分けて2文で答え、どちらの日付が正しいか推測しないでください。Inspect the data and repeat the comparison
Download the protocol and all 18 responses as JSON. The file includes registration time, task criteria, requested and returned model IDs, settings and complete output text.
Agreement on simple synthetic tasks does not establish agreement on complex work, long context, current research or tool use. More repeats, other tasks and matched model-selection criteria would be needed for broader conclusions. We did not measure latency, cost-effectiveness or long-term reliability; token-usage records alone should not be turned into those rankings.
For your own comparison, define what matters first, preserve the inputs and outputs, and verify the answer in the intended setting. Chappie’s comparison workflow is one option for comparing model answers in a shared conversation. This API experiment does not guarantee app output, app performance or the models available to an app customer.
See the ChatGPT, Gemini and Claude buying guide or choosing a Poe alternative for product and subscription decisions.
