Tokens in Other Languages

Part of the Fundamentals section of Coddy's AI Prompts journey — lesson 11 of 23.

Most AI models were trained primarily on English text. This has a surprising consequence: non-English languages often use more tokens for the same meaning.

The word hello in English is typically one token. But its equivalents in other languages might be split differently:

Language	Word	Approximate Tokens
English	`hello`	1
Spanish	`hola`	1
Japanese	`こんにちは`	3-5
Arabic	`مرحبا`	2-4
Russian	`привет`	2-3

Languages with non-Latin scripts—like Chinese, Japanese, Korean, Arabic, or Hindi—tend to use significantly more tokens. A sentence that takes 10 tokens in English might require 20 or 30 tokens in Japanese.

This matters for practical reasons. If you're working in a non-English language, you'll hit context window limits faster, responses may cost more, and generation might take slightly longer. It's not that the AI is worse at these languages—it simply processes them less efficiently at the token level.

Cheat sheet

AI models trained primarily on English text use tokens less efficiently for non-English languages. Non-Latin scripts (Chinese, Japanese, Korean, Arabic, Hindi) typically require significantly more tokens for the same meaning.