Words ≠ Tokens

Part of the Fundamentals section of Coddy's AI Prompts journey — lesson 8 of 23.

A common mistake is assuming that one word equals one token. It doesn't work that way.

The word cat is one token. But cats? That might be two tokens: cat and s. The word hello is one token, but Hello with a capital H could be tokenized differently.

Spaces, punctuation, and capitalization all affect how text gets split.

Here are some surprising examples:

Text	Approximate Tokens
`dog`	1
`dogs`	1-2
`extraordinary`	2-3
`123456`	2-3
`!!!`	1-3

Numbers are especially tricky. You might think 1000 is one token, but it could be split into 100 and 0, or even digit by digit. This is why AI sometimes struggles with precise math—it's not seeing numbers the way you do.

The key insight: token counts are unpredictable if you're just counting words. A 100-word prompt might use 130 tokens, or 150, depending on the specific words you chose.

Watch the token counter in the chat panel as you type—you'll see it doesn't increase by exactly one per word.

Cheat sheet

Tokens are not the same as words. Tokenization is affected by spaces, punctuation, and capitalization.

Examples of how text gets tokenized:

Text	Approximate Tokens
`cat`	1
`cats`	1-2 (may split into `cat` and `s`)
`hello` vs `Hello`	May tokenize differently due to capitalization
`extraordinary`	2-3
`123456`	2-3
`!!!`	1-3

Numbers can be split unpredictably (e.g., 1000 might become 100 and 0, or split digit by digit), which is why AI sometimes struggles with precise math.

Key insight: Token counts are unpredictable when counting words. A 100-word prompt might use 130-150 tokens depending on the specific words chosen.

Try it yourself

This lesson doesn't include a code challenge.