What Are Tokens?

AI breaks your text into smaller pieces called tokens.

Think of tokens as the AI's basic unit of reading. Just like you might break a sentence into words, the AI breaks text into tokens.

But here's the twist: tokens aren't always whole words. A token might be a word, part of a word, or even just a single character.

For example, the word hello is typically one token. But a longer word like uncomfortable might be split into multiple tokens: un, comfort, able. The AI processes each token separately, one after another.