What Are Tokens?
Part of the Fundamentals section of Coddy's AI Prompts journey — lesson 7 of 23.
AI breaks your text into smaller pieces called tokens.
Think of tokens as the AI's basic unit of reading. Just like you might break a sentence into words, the AI breaks text into tokens.
But here's the twist: tokens aren't always whole words. A token might be a word, part of a word, or even just a single character.
For example, the word hello is typically one token. But a longer word like uncomfortable might be split into multiple tokens: un, comfort, able. The AI processes each token separately, one after another.
Challenge
Get the AI to respond with exactly 4 words or less, where each word has only one syllable. The response should be a complete thought or statement.
The AI's response must:
- Contain 4 words (or less)
- Not contain any punctuation except for a single period at the end
Cheat sheet
AI breaks text into smaller pieces called tokens, which are the AI's basic unit of reading.
Tokens aren't always whole words. A token might be:
- A complete word (e.g.,
hello) - Part of a word (e.g.,
uncomfortablemight split intoun,comfort,able) - A single character
The AI processes each token separately, one after another.
Start this challenge with Coddy's AI tutor
Test yourselfThis lesson includes a short quiz. Start the lesson to answer it and track your progress.
All lessons in Fundamentals
2How AI Reads You — Tokens
What Are Tokens?Words ≠ TokensWhy Token Counts MatterThe Context WindowTokens in Other LanguagesSpecial Tokens You Never See3Writing Your First Prompts
AI Talks Markdown (By Default)Controlling the Output FormatControlling Tone and StyleWhen the AI Ignores You