Chain of thought prompting asks a language model to write out the intermediate steps of a problem before it gives the final answer. On problems that take several steps, such as word problems, logic puzzles and schedules, working in the open tends to produce more correct answers than answering at once, and it leaves you steps you can check. The shortest version is a single sentence added to the prompt: "Let's think step by step."
Why writing the steps helps
A model produces its reply one token at a time, and everything it has already written becomes input for the next token. If a prompt demands the answer immediately, the model has to produce it before any intermediate result exists on the page. If it first writes "Monday's coffee sales came to $168", that number is now in the context, and the next step can build on it instead of holding it implicitly.
That is the intuition. The evidence came from two 2022 papers. Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", showed that worked examples with their reasoning written out improved large models on arithmetic, commonsense and symbolic reasoning tasks. They also found the benefit depended on scale: smaller models did not gain, and often wrote fluent reasoning that led to wrong answers. Kojima et al., "Large Language Models are Zero-Shot Reasoners", then showed that one sentence with no examples, "Let's think step by step", also improved results substantially, though generally less than worked examples did.
Direct answer versus chain of thought
Both tabs ask the same question. The first demands the amount only; the second asks for the working and puts the answer on a fixed last line.
$258
The direct reply shows a typical slip: 168 of coffee plus Tuesday's $90 of muffins, so it forgot that Tuesday sold 10 fewer coffees. The step-by-step reply gives Tuesday's coffee count its own line, so that step cannot be skipped silently. A current model may get the direct version right too; the gap widens as problems get longer and the steps depend on each other more.
The written steps have a second benefit: when an answer is wrong, you can see which step broke, and you can fix the prompt or the input at that point.
Few-shot chain of thought
The original technique from Wei et al. puts worked examples in the prompt whose answers show their reasoning, and the model answers the new question the same way. The example below is adapted from the first figure of their paper; the question after it is new. Change the question to try your own.
The library started with 120 books. Lending out 45 left 75 on the shelves. Getting 18 back made 93. The donation of 30 brought it to 123. The answer is 123.
Few-shot CoT gives you control over the shape of the reasoning: how long it is, what it spells out and how the answer is stated. "The answer is 11" in the example is a fixed ending, and the reply copied it. Zero-shot CoT is faster to write but leaves those choices to the model. For more on building example sets, see few-shot prompting; for the instruction-only version, see zero-shot prompting.
Put the answer on its own line
Once the reply contains reasoning, the answer is somewhere inside a paragraph, which is a problem when a program has to read it. Ask for the answer in a fixed form on the last line, as the café prompt does with "Answer: $X", and read that line in code:
import re
matches = re.findall(r"^Answer:\s*(.+)$", reply, re.MULTILINE)
answer = matches[-1] if matches else None
Taking the last match matters, because the reasoning itself may contain a line starting with "Answer:" before the model corrects itself. If users should see only the answer, ask for the reasoning inside one pair of tags and the answer inside another, then show only the second. Structured output covers asking for JSON when you need more than one field.
Reasoning models
Some current models are built to think before they answer: they generate reasoning internally, often hidden or summarized, before the visible reply. For them, "Let's think step by step" matters much less, because the steps happen whether you ask or not. Adding it usually just makes the visible reply longer.
What still helps with a reasoning model is everything around the reasoning: a complete statement of the problem, the constraints a correct answer must satisfy, and the format of the final answer. Describe what a good answer looks like rather than scripting each step; a fixed recipe of steps may steer the model away from a better route it would have found on its own.
When not to use chain of thought
Chain of thought costs tokens and time, and it only pays off when a task has steps that depend on each other. For a lookup, a translation, a rewrite or a simple classification, it just makes the reply longer. A model can also write reasoning that looks sound and still reach a wrong answer, so check the final result on its own, not only the steps that led to it.
For hard problems where a single chain may go wrong, two extensions build on it. Self-consistency prompting samples several chains and takes the answer most of them reach, and tree of thought prompting explores and compares several partial lines of reasoning before committing to one.
Frequently Asked Questions
What is chain of thought prompting?
Chain of thought (CoT) prompting asks a language model to write out the intermediate steps of a problem before its final answer, either by showing worked examples that include their reasoning or by adding an instruction such as "Let's think step by step." On multi-step problems this tends to produce more correct answers, and it lets you check each step.
Does "let's think step by step" still work?
With standard chat models it still helps on problems that take several steps, especially when the prompt would otherwise push for an instant answer. Reasoning models, which think before they reply, already do this internally, so the phrase adds little for them beyond a longer reply. For those models, state the problem and the answer format clearly instead.
When does chain of thought help, and when does it not?
It helps on tasks that need several dependent steps: word problems, arithmetic, logic puzzles, scheduling with constraints and tracing what code does. It adds little to lookups, translation, rewriting or simple classification, where it only makes the reply longer and slower.
What is the difference between zero-shot and few-shot chain of thought?
Few-shot CoT, introduced by Wei et al. in 2022, puts worked examples with their reasoning in the prompt, and the model imitates that style of reasoning. Zero-shot CoT, from Kojima et al. 2022, uses no examples and simply adds an instruction such as "Let's think step by step." Zero-shot is quicker to write; few-shot gives more control over how the steps look.
Is the written reasoning how the model actually got the answer?
Not necessarily. The steps are text the model generates, and they can read as sound while the final answer came out wrong, or contain an error while the answer happens to be right. Treat the reasoning as something to check, not as proof, and verify the final answer on its own.
