Study · 4 variations
Best country to live in? (varied language)
The same two prompts asked in English, Mandarin Chinese, Hindi and Spanish. Answers in other languages are translated to English country names during normalization, so the four variations share one set of answers. The overall goal is to see whether the language of the question changes the answer in a meaningful way.
Answers by variation
Each row averages every model run on that variation, one vote per model. Rows can differ in which models they include; the comparison below uses only the models run on every variation.Largest differences
How often an answer is given in one variation, in percentage points, against the average of the other variations, over the 83 models run on all of them.- canada +6.1 pts in Hindi vs the other variations · 83 models
- switzerland +5.4 pts in English vs the other variations · 83 models
- canada −4.6 pts in English vs the other variations · 83 models
- switzerland −4.1 pts in Spanish vs the other variations · 83 models
- switzerland −3.5 pts in Hindi vs the other variations · 83 models
- canada −3.3 pts in Mandarin vs the other variations · 83 models
- portugal −2.9 pts in Hindi vs the other variations · 83 models
- portugal +2.6 pts in English vs the other variations · 83 models
Per model
Sorted by spread, the largest distance between any two variations the model was run on. A dot marks a variation the model hasn't been run on. Hover a bar for its breakdown.| Model | English | Mandarin | Hindi | Spanish | Spread |
|---|---|---|---|---|---|
| anthropic/claude-haiku-4.5 | | | | | 100 |
| anthropic/claude-opus-4.6 | | | | | 100 |
| anthropic/claude-sonnet-4.6 | | | | | 100 |
| google/gemini-2.5-flash | | | | | 100 |
| ibm-granite/granite-4.1-8b | | | | | 100 |
| minimax/minimax-m2.5 | | | | | 100 |
| mistralai/mistral-small-2603 | | | | | 100 |
| mistralai/mistral-small-3.2-24b-instruct | | | | | 100 |
| qwen/qwen3-235b-a22b-2507 | | | | | 100 |
| x-ai/grok-4-fast | | | | | 100 |
| xiaomi/mimo-v2-omni | | | | | 95 |
| minimax/minimax-m2.1 | | | | | 93.3 |
| anthropic/claude-sonnet-5 | | | | | 90 |
| deepseek/deepseek-v3.2 | | | | | 90 |
| inception/mercury-2 | | | | | 90 |
| openai/gpt-4o-mini | | | | | 90 |
| upstage/solar-pro4 | | | | | 90 |
| openai/gpt-5.4-nano | | | | | 76.6 |
| qwen/qwen3.6-flash | | | | | 75 |
| x-ai/grok-4.20 | | | | | 75 |
+ 112 more models hide
| upstage/solar-pro-3 | | | | | 70 |
| openai/gpt-oss-120b | | | | | 65 |
| stepfun/step-3.7-flash | | | | | 65 |
| anthropic/claude-fable-5 | | | | | 60 |
| nvidia/nemotron-3-ultra-550b-a55b | | | | | 60 |
| qwen/qwen3.6-plus | | | | | 60 |
| sakana/fugu-ultra | | | | | 60 |
| google/gemini-2.5-flash-lite | | | | | 55 |
| anthropic/claude-sonnet-4.5 | | | | | 50 |
| google/gemini-3-flash-preview | | | | | 50 |
| google/gemma-4-31b-it | | | | | 50 |
| mistralai/mistral-nemo | | | | | 50 |
| moonshotai/kimi-k3 | | | | | 50 |
| upstage/solar-mini4 | | | | | 50 |
| xiaomi/mimo-v2.5-pro | | | | | 50 |
| z-ai/glm-4.7-flash | | | | | 50 |
| minimax/minimax-m2.7 | | | | | 49 |
| qwen/qwen3.6-27b | | | | | 48 |
| z-ai/glm-5.2 | | | | | 46.6 |
| anthropic/claude-opus-5 | | | | | 45 |
| moonshotai/kimi-k2.7-code | | | | | 45 |
| openai/gpt-4.1-mini | | | | | 45 |
| openai/gpt-5.4-mini | | | | | 45 |
| openai/gpt-5.5 | | | | | 45 |
| xiaomi/mimo-v2-pro | | | | | 45 |
| xiaomi/mimo-v2.5 | | | | | 45 |
| z-ai/glm-4.7 | | | | | 44.5 |
| qwen/qwen3.7-max | | | | | 41.7 |
| moonshotai/kimi-k2.6 | | | | | 40 |
| openai/gpt-5.6-terra | | | | | 40 |
| openai/gpt-6-luna | | | | | 40 |
| openai/gpt-6-sol | | | | | 40 |
| tencent/hy3:free | | | | | 40 |
| z-ai/glm-5 | | | | | 38.3 |
| openai/gpt-5.3-chat | | | | | 37.4 |
| minimax/minimax-m3 | | | | | 35 |
| x-ai/grok-4.5 | | | | | 35 |
| deepseek/deepseek-v4-flash | | | | | 33.4 |
| x-ai/grok-4.3 | | | | | 33.4 |
| z-ai/glm-5.1 | | | | | 31 |
| anthropic/claude-opus-5.5 | | | | | 30 |
| google/gemma-4-26b-a4b-it | | | | | 30 |
| openai/gpt-5.6-luna | | | | | 30 |
| x-ai/grok-4.1-fast | | | | | 30 |
| qwen/qwen3.6-max-preview | | | | | 26.7 |
| qwen/qwen3.7-plus | | | | | 26.7 |
| deepseek/deepseek-v4-pro | | | | | 20 |
| moonshotai/kimi-k2.5 | | | | | 20 |
| openai/gpt-6.1-sol | | | | | 20 |
| thinkingmachines/inkling | | | | | 20 |
| x-ai/grok-4.7 | | | | | 20 |
| qwen/qwen3.5-flash-02-23 | | | | | 13.4 |
| z-ai/glm-5-turbo | | | | | 13.3 |
| anthropic/claude-sonnet-5.5 | | | | | 10 |
| inception/mercury-2.5 | | | | | 10 |
| openai/gpt-5.6-sol | | | | | 10 |
| anthropic/claude-opus-4.8 | | | | | 6.7 |
| qwen/qwen3.5-122b-a10b | | | | | 6.7 |
| muse-spark-1.1 | | | | | 5 |
| anthropic/claude-opus-4.7 | | | | | 0 |
| google/gemini-3.1-flash-lite | | | | | 0 |
| google/gemini-3.5-flash | | | | | 0 |
| openai/gpt-5.4 | | | | | 0 |
| amazon/nova-2-lite-v1 | | · | · | · | — |
| amazon/nova-lite-v1 | | · | · | · | — |
| amazon/nova-micro-v1 | | · | · | · | — |
| anthropic/claude-3-haiku | | · | · | · | — |
| anthropic/claude-fable-5.1 | | · | · | · | — |
| bytedance-seed/seed-1.6 | | · | · | · | — |
| bytedance-seed/seed-1.6-flash | | · | · | · | — |
| bytedance-seed/seed-2-1-turbo | | · | · | · | — |
| bytedance-seed/seed-2.0-lite | | · | · | · | — |
| deepseek/deepseek-chat-v3-0324 | | · | · | · | — |
| deepseek/deepseek-r1 | | · | · | · | — |
| deepseek/deepseek-v4-flash-0731 | | · | · | · | — |
| deepseek/deepseek-v4-pro-0813 | | · | · | · | — |
| google/gemini-3.6-flash | | · | · | · | — |
| google/gemini-3.7-flash | | · | · | · | — |
| google/gemini-3.8-flash | | · | · | · | — |
| google/gemma-2-27b-it | | · | · | · | — |
| meta-llama/llama-3.1-70b-instruct | | · | · | · | — |
| meta-llama/llama-3.1-8b-instruct | | · | · | · | — |
| meta-llama/llama-3.3-70b-instruct | | · | · | · | — |
| meta-llama/llama-4-maverick | | · | · | · | — |
| meta-llama/llama-4-scout | | · | · | · | — |
| meta/muse-glimmer-30b | | · | · | · | — |
| meta/muse-spark-1.1 | | · | · | · | — |
| meta/muse-spark-1.2 | | · | · | · | — |
| meta/muse-spark-1.3 | | · | · | · | — |
| mistralai/mistral-large | | · | · | · | — |
| mistralai/mistral-small-24b-instruct-2501 | | · | · | · | — |
| nvidia/nemotron-3.5-lightning | | · | · | · | — |
| openai/gpt-3.5-turbo | | · | · | · | — |
| openai/gpt-4 | | · | · | · | — |
| openai/gpt-4.1 | | · | · | · | — |
| openai/gpt-4.1-nano | | · | · | · | — |
| openai/gpt-5.2 | | · | · | · | — |
| openai/gpt-6-astra | | · | · | · | — |
| openai/o3 | | · | · | · | — |
| openai/o3-mini | | · | · | · | — |
| openai/o4-mini | | · | · | · | — |
| qwen/qwen3.7-flash | | · | · | · | — |
| qwen/qwen3.8-27b | | · | · | · | — |
| qwen/qwen3.8-flash | | · | · | · | — |
| qwen/qwen3.8-max-0902 | | · | · | · | — |
| tencent/hy-mt2-30b-a3b | | · | · | · | — |
| tencent/hy3 | | · | · | · | — |
| tencent/hy4-preview | | · | · | · | — |
| thinkingmachines/inkling-small | | · | · | · | — |
| x-ai/grok-4.6 | | · | · | · | — |
| z-ai/glm-5.3 | | · | · | · | — |
| z-ai/glm-5.3-flash | | · | · | · | — |