How many more tokens does Thai take than English?
The same article in Thai versus English, measured on 109 stories this site publishes in both languages.
- Qwen3.5 – 3.8 1.20×
- Gemini 2.5 · 3.x ≈1.40×
- Llama 4 ≈1.43×
- Grok 4.6 · 4.7 ≈1.44×
- Amazon Nova 2 ≈1.53×
- DeepSeek V4 · V4.1 ≈1.57×
- GPT-4o · 4.1 · 5.x · 6 1.67×
- Llama 3.3 ≈1.83×
- Mistral Medium 3.5 ≈1.96×
- Typhoon 2.5 · Qwen3 1.99×
- Claude Opus · Sonnet · Fable 5.x ≈2.03×
- Claude Haiku 4.5 ≈2.73×
- Kimi K2.5 · K2.6 ≈2.73×
- Kimi K3 ≈2.76×
- GLM-5.3 ≈3.03×
- MiniMax M2.7 · M3 ≈3.47×
≈ = derived through a ratio measured against GPT (see method below)
- Typhoon 2.5, a Thai model, uses Qwen3’s tokenizer unchanged — the file is byte-identical — so a Thai article costs 1.99× its English version — a wider gap than GPT’s 1.67×.
- Qwen3.5 through 3.8 share a new tokenizer (248,000-entry vocabulary, up from 151,643); the Thai–English gap drops to 1.20×.
- GPT-6 (Astra, Sol, Luna) and GPT-5.6 use the same tokenizer as GPT-5, o200k — measured through OpenRouter, the counts matched on every text — so they are counted exactly in your browser.
- On the same Thai text, these use fewer tokens than GPT (× GPT in brackets): Qwen3.5 – 3.8 (0.77), Llama 4 (0.84), Amazon Nova 2 (0.88), Grok 4.6 · 4.7 (0.89), Gemini 2.5 · 3.x (0.92), DeepSeek V4 · V4.1 (0.94).
- The most tokens on Thai text (× GPT): MiniMax M2.7 · M3 (2.03), Claude Opus · Sonnet · Fable 5.x (1.99), Claude Haiku 4.5 (1.87).
- Anthropic says Claude’s new tokenizer (4.7 onward) counts about 30% more. Measured: +43% on English, only +6% on Thai.
How this was measured
- GPT, Typhoon and Qwen are counted with the real tokenizers in your browser, tested against the reference libraries (tiktoken and Hugging Face tokenizers) on 33 real texts with zero mismatches.
- Claude, Gemini, Grok, DeepSeek, Llama, Mistral, Kimi, GLM, MiniMax and Nova have no tokenizer we can run in your browser. We sent 12 article pairs through OpenRouter, which reports each provider’s own token count, and measured the ratio to GPT separately for Thai and English. Estimates here are GPT tokens × that ratio, with the p10–p90 spread shown as a range.
- Models from one vendor share a row only when they measured identical on every text — Gemini 3.8 Flash and 2.5 Flash-Lite, for example. DeepSeek through OpenRouter sometimes reports exactly 79 extra tokens (a provider adds its own text); otherwise every V4 model counts the same.
- Left out: GPT-6 Pro variants report 260–470 more input tokens than the base model, varying by text, so their cost cannot be computed from a token count. GLM-5.2 and Amazon Nova Pro counted differently from their measured siblings. Mistral Small and Cohere refused every measurement request.
- For exact Claude counts, Anthropic’s count_tokens API is free to use (API key required).
- Prices are pulled from OpenRouter every time the site rebuilds (daily). Latest: 2026-09-23.
FAQ
What is a token?
The unit a model splits text into — a whole word, part of a word, or a single character. API billing and context limits are counted in tokens, not characters.
Is my text sent anywhere?
No. Counting happens entirely in your browser; the tokenizer files load from this site and your text is never transmitted.
Why does Thai use more tokens?
Most tokenizers are trained mainly on English, so Thai words get split into smaller pieces — and Thai has no spaces between words, which makes splitting harder still.
How can I cut Thai token costs?
Pick a model whose tokenizer handles Thai well (Qwen3.5 – 3.8 and Llama 4 use the fewest of those measured), write long reused system prompts in English, and use prompt caching for content you send every time.