Self-consistency prompting means asking a model the same reasoning question several times and keeping the answer that comes up most often. Each run writes out its own reasoning, as in chain of thought prompting, and because the model samples its words with some randomness, the runs take different paths. Correct paths tend to arrive at the same answer. Mistakes tend to scatter across different wrong ones. A majority vote exploits that difference.
The technique comes from Wang et al. 2022, "Self-Consistency Improves Chain of Thought Reasoning in Language Models". The paper compared sampling many reasoning paths and voting against taking a single greedy answer, and found that voting improved accuracy on arithmetic and commonsense reasoning benchmarks.
The block below shows three runs of the same prompt. Each tab is one run with its own reply. Two runs count correctly; one makes a slip that is easy to make.
Count the 7s by position.
- Ones place: 7, 17, 27, 37, 47, 57, 67, 77, 87, 97. That is 10 sevens.
- Tens place: 70 to 79. That is 10 more sevens.
- Hundreds place: only 100 has three digits, and it has no 7.
77 appears in both lists, which is right, because it contains two 7s.
Answer: 20
Run 2 answered a slightly different question: how many numbers contain a 7, not how many times the digit appears. Its reasoning reads just as confident as the other two, and on its own you would have no reason to doubt it. Across three runs, the vote is 20, 19, 20, and the majority answer is the correct one.
How self-consistency works
- Write a chain-of-thought prompt with a fixed answer line. Ask for reasoning first and the final answer last, in a form you can find, such as "Answer: N". Without a fixed line, you have to read every run to find its answer.
- Sample several runs with randomness on. Each run needs the freedom to take a different path. In the API, a temperature above 0 does this. At temperature 0 most runs would repeat the same reasoning, and repeating a mistake five times is not evidence.
- Extract the final answer from each run. Ignore the reasoning at this stage. Two runs that reached 20 by different routes count as two votes for 20.
- Take the most common answer. Normalize before counting, so that "20", "20." and "twenty" count as the same answer.
The agreement itself is useful information. When all five runs agree, a random slip is unlikely. When the votes split three ways, the question is hard for the model, and that is a signal to check the answer yourself or to rewrite the prompt. The paper also reported that questions with stronger agreement between samples were more likely to be answered correctly.
Self-consistency by hand
You can apply the technique in any chat app. Send the prompt, note the final line, then get another independent run: press regenerate, or paste the same prompt into a new chat. Repeat until you have three or five answers.
Do not ask again in the same conversation with "Are you sure?" or "Try again". The model can see its previous reply, so the new answer is not independent: it tends either to repeat the first answer or to change it just because you seemed to doubt it. Neither is a vote.
This prompt is set up for voting. Put in your own question and answer format, then run it several times.
12 pens is 12 ÷ 3 = 4 packs.
4 packs cost 4 × 8.
Change from 10 minus 2.
Answer: $2
The format part is what makes the vote possible. Switch it off and the runs tend to phrase their answers differently, often somewhere in the middle of the text, which turns a simple count into careful reading.
Self-consistency in code
In code, the loop is short: run the same prompt N times, pull out the answer lines, and count them. This version uses the OpenAI Python SDK; any API that lets you set a temperature works the same way.
import re
from collections import Counter
from openai import OpenAI
client = OpenAI()
MODEL = "your-model-id" # e.g. from your provider's model list
PROMPT = (
"How many times does the digit 7 appear when you write out all the whole "
"numbers from 1 to 100? Think it through step by step, then give the final "
'answer on the last line in the form "Answer: N".'
)
def final_answer(text):
# Also matches "**Answer: 20**", since chat models often bold the last line.
lines = re.findall(r"^\**Answer:\**\s*(.+)$", text, re.MULTILINE)
return lines[-1].strip(" *.") if lines else None
answers = []
for _ in range(5):
response = client.chat.completions.create(
model=MODEL,
temperature=0.8,
messages=[{"role": "user", "content": PROMPT}],
)
answers.append(final_answer(response.choices[0].message.content))
votes = Counter(a for a in answers if a is not None)
if votes:
best, count = votes.most_common(1)[0]
print(f"{best} ({count} of {len(answers)} runs agree)")
else:
print("No run gave an answer line.")
The runs are independent, so in production you would send them in parallel rather than one after another. Some reasoning models accept only their default temperature and return an error if you set one; for those, leave temperature out. Their runs still vary, because the default already samples.
When to use it, and what it costs
Self-consistency is worth it when three things are true: the answer is short and comparable (a number, a label, a choice), a wrong answer costs more than a few extra calls, and the model is not already reliable on the task. Math word problems, classification with tricky edge cases and multiple-choice questions fit well.
It costs roughly N times the tokens of a single answer, and it does not help when the model shares the same misconception on every run. If all five runs make the same mistake, the vote is unanimous and wrong. Voting reduces random slips, not systematic errors, so it is no substitute for checking the answers against a known result now and then.
Models that reason internally before answering already work through the problem at length on every run, which tends to shrink the gain from voting. It can still pay off on hard questions where their runs disagree.
Self-consistency votes only on finished answers. Tree of thought prompting goes a step further and compares partial reasoning while the problem is still being solved, keeping promising branches and dropping weak ones before they reach an answer.
Frequently Asked Questions
What is self-consistency prompting?
Self-consistency prompting is a technique from Wang et al. (2022). You send the same chain-of-thought prompt several times with sampling turned on, so each run reasons along a different path, then you take the final answer that appears most often. It replaces trusting one reasoning chain with a vote across several.
How many samples should I use for self-consistency?
For everyday use, three to five runs are enough to outvote an occasional slip, and an odd number avoids ties between two answers. The original paper sampled many more paths per question and found that accuracy kept rising with more samples but with smaller gains each time. Use more runs when a wrong answer is expensive and fewer when cost or speed matters.
What is the difference between self-consistency and chain of thought?
Chain of thought is one run in which the model writes out its reasoning before the answer. Self-consistency builds on it: it runs chain of thought several times and votes on the final answers. Chain of thought changes the prompt; self-consistency changes how many answers you collect and how you pick one.
Does self-consistency work for essays or open-ended answers?
Not directly, because a vote needs answers that can be compared for equality: a number, a label, a multiple-choice letter or a short fact. For open-ended text, a common workaround is to generate several versions and ask the model which one agrees best with the others, but that is a judgement, not a count.
Can I use self-consistency in ChatGPT or Claude?
Yes, by hand. Send the prompt in several new chats, or regenerate the reply several times, and write down the final answer from each run. Use a fresh chat or the regenerate button rather than asking again in the same thread, because a model that can see its previous answer tends to repeat it.
