Tokens and Context Window: How AI Reads Text

AI language models do not read letters or words. They read tokens: small chunks of text, often a whole word, sometimes part of one. And they can only take in a limited number of tokens at a time, a limit called the context window. Those two ideas explain why AI tools are priced the way they are, why long chats start to forget early instructions, and why a model can struggle to count the letters in a word it just spelled correctly.

What is a token in AI?

Before a model sees your prompt, a program called a tokenizer splits the text into pieces from a fixed vocabulary and turns each piece into a number. The model works entirely with those numbers, and it writes its reply the same way, one token at a time, which the app turns back into text.

The vocabulary is learned from large amounts of text, so common words tend to be a single token and rarer words are split into several familiar pieces. A space is often attached to the start of the word after it. An English sentence might be split roughly like this:

Token ization isn 't magic .

That split is illustrative. Each model family has its own tokenizer, and the same sentence can come out as a different number of pieces in each. Many providers publish a tokenizer tool or a token counting API so you can check real counts.

Images, audio and files are converted into tokens too when a model accepts them, which is why an attached screenshot uses up part of the same budget as your text.

How many tokens is a word?

For English text, a widely used rule of thumb is about four characters per token, or roughly three quarters of a word. By that estimate, 1,000 tokens is around 750 English words, and a 3,000-word article is around 4,000 tokens.

The ratio shifts with the content:

• Other languages often take more tokens for the same meaning. Tokenizers are trained on data where English is heavily represented, so English words get compact tokens while text in Japanese, Korean, Arabic, Hindi and many other languages is split into more pieces. The gap depends on the tokenizer and has narrowed in newer ones, but it has not disappeared.
• Code uses tokens on indentation, brackets, operators and long identifiers, so a file often costs more tokens than its word count suggests.
• Numbers, URLs and unusual strings such as IDs and hashes tend to be split into many small tokens.

Why tokens matter

Cost. APIs charge per token, with separate prices for the tokens you send and the tokens the model writes. In a chat, the whole conversation is sent again with every new message, so a long conversation costs more per message than a short one.

Limits. A model has a context window for everything it reads and writes in one request, and often a separate cap on how long a single reply can be. In the API you can set that cap yourself, and with Anthropic's API you must: max_tokens is a required parameter.

Speed. The reply is produced one token after another, so a longer reply takes proportionally longer to finish.

Spelling tasks. Since a model sees a word like magic as one or two units rather than five letters, tasks that depend on individual characters, such as counting letters, reversing a word or finding words of an exact length, are harder for it than they look. Newer models handle many of these better, but when the characters matter, check the answer or ask the model to write the word out one letter at a time first.

What is a context window?

The context window is the maximum number of tokens a model can take into account in a single request. Think of it as the model's working memory: everything the model can use to write its reply has to fit inside it at once, including:

• the system prompt, the app's own instructions and yours
• the conversation so far, every user message and every reply
• attached files, pasted documents and search or tool results
• the reply being written

Anything outside the window does not exist for the model. There is no background memory it can look things up in. Context window sizes vary widely between models and keep growing, so check your provider's documentation for the one you use rather than relying on a number from an article.

The model also keeps nothing between requests. What looks like memory in a chat is the app sending the whole conversation again each time. Memory features in chat apps work the same way: the app saves notes about you, or searches your earlier chats, and inserts what it finds into the context of new chats.

What happens when a chat gets long

When a conversation outgrows the context window, something has to give. Depending on the app, the oldest messages are dropped, older parts are replaced with a summary, or you are told the chat has hit its limit and should start a new one. In each case, details from the beginning, including instructions you gave in the first message, can stop affecting the answers.

A full window is not the only problem. The more material a model has in front of it, the more there is to overlook, and in a long chat, the early decisions compete with everything said since. Symptoms include answers that ignore a constraint you set earlier, code that reverts to a version you already fixed, or a model that repeats an idea you rejected.

Working within the context window

Start a new chat for each task. A fresh chat with a clear first message usually beats a long chat that has drifted through several topics.

Carry a summary forward. When a chat has built up useful decisions, ask for a summary written to be pasted into a new chat, then continue there. Fill in the blanks below to fit your work.

Continue in a new chat

EditCopy

Fill inartifactwords

This chat is getting long. Write a summary I can paste into a new chat to continue the work. Include: the goal, the decisions we made and why, the current version of the database schema, and the open questions. Keep the summary under 200 words, not counting the database schema itself.

Try it

Example replyReplies vary between models and runs.

Goal: a database for a small library app, stored in SQLite.

Decisions:

• Loans get their own table, so the full lending history is kept.
• A book counts as available when it has no loan with an empty returned_at.

Current schema:

CREATE TABLE books (id INTEGER PRIMARY KEY, title TEXT NOT NULL, author TEXT NOT NULL);
CREATE TABLE members (id INTEGER PRIMARY KEY, name TEXT NOT NULL, email TEXT UNIQUE);
CREATE TABLE loans (
  id INTEGER PRIMARY KEY,
  book_id INTEGER NOT NULL REFERENCES books(id),
  member_id INTEGER NOT NULL REFERENCES members(id),
  loaned_at TEXT NOT NULL,
  returned_at TEXT
);

Open questions:

• Do we need late fees?
• Should the library track several copies of the same book?

Paste only what the task needs. The function with the bug and the code that calls it, not the whole repository. The section of the contract in question, not all forty pages. Less material leaves less to overlook.

Put the question after long material, and restate what matters. When you paste a long document, ask your question at the end, after it, and repeat the key constraints there. The instruction then sits next to the point where the model starts writing.

Split a document that is too long for one message. Some apps limit how much you can paste into a single message even when the model's window could hold more. Sending the document in parts gets around the message limit, but not the context window: every part still counts toward it. Tell the model up front not to respond until you have sent everything.

Send each partAsk the question

EditCopy

Fill intotalquestion

I am going to send you a long document in 3 parts. After each part, reply only with "Received part N of 3" and nothing else. Do not summarize or answer anything until I ask my question. Part 1 of 3: """ [paste part 1] """

Try it

Example replyReplies vary between models and runs.

Received part 1 of 3

Count tokens when it matters. If you are building with an API, count before you send. OpenAI's open-source tiktoken library tokenizes text with OpenAI's tokenizers, and Anthropic's API has a token counting endpoint. Counts from one provider's tokenizer are only an estimate for another's.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")  # one of OpenAI's tokenizers
tokens = enc.encode("Tokenization isn't magic.")
print(len(tokens))
print([enc.decode([t]) for t in tokens])

When you build an app around a model, deciding what goes into the window and in what order becomes a design problem of its own. Context engineering covers that, and prompt chaining shows how to split a large job into steps that each fit comfortably. For how a single message fits into the whole input, see what is a prompt.