Menu
CoddyTech

Tokens in Other Languages

Part of the Fundamentals section of Coddy's AI Prompts journey. Lesson 11 of 23.

Most AI models were trained primarily on English text. This has a surprising consequence: non-English languages often use more tokens for the same meaning.

The word hello in English is typically one token. But its equivalents in other languages might be split differently:

LanguageWordApproximate Tokens
Englishhello1
Spanishhola2
Japaneseこんにちは1
Arabicمرحبا3-5
Russianпривет3-4

A very common word like こんにちは can still be a single token, because it appeared often in the text the tokenizer was built from.

Languages with non-Latin scripts, like Chinese, Japanese, Korean, Arabic, or Hindi, tend to use significantly more tokens. A sentence that takes 10 tokens in English might require 20 or 30 tokens in Japanese.

This matters for practical reasons. If you're working in a non-English language, you'll hit context window limits faster, responses may cost more, and generation might take slightly longer. Using more tokens does not by itself make the AI worse at a language, but models trained mostly on English text often do give weaker answers in languages that were less common in their training data.

Try it yourself

This lesson doesn't include a code challenge.

quiz iconTest yourself

This lesson includes a short quiz. Start the lesson to answer it and track your progress.

All lessons in Fundamentals