Tree of Thought Prompting: How It Works, Examples

Tree of thought prompting makes a language model work on a problem the way you might on paper: write down a few possible next steps, judge which ones look promising, continue those, and abandon a branch when it leads nowhere. The idea comes from Yao et al. 2023, "Tree of Thoughts: Deliberate Problem Solving with Large Language Models". It extends chain of thought prompting, which follows a single line of reasoning, into a search over many lines.

This page explains how the original method works, gives you a one-prompt version to try in ChatGPT, Claude or Gemini, and shows the loop in code for when one prompt is not enough.

How tree of thoughts works

The paper breaks the method into four decisions.

1. What counts as a thought. A thought is one intermediate step, small enough for the model to produce well and big enough to judge. In a math puzzle it is one equation; in a writing task it is a short plan.
2. How to generate thoughts. From the current partial solution, the model proposes several candidate next steps, either by sampling independently or by listing them in one reply.
3. How to evaluate them. The model is asked to rate each partial solution. The paper used two styles: score each state on its own (for example as "sure", "likely" or "impossible"), or show the model several states and let it vote for the best.
4. How to search. A program keeps the best few states at each level (breadth-first search) or follows one branch deep and backs up when the evaluation says it is hopeless (depth-first search).

Take the paper's Game of 24 task: use the numbers 4, 9, 10 and 13 once each, with the four basic operations, to make 24. A chain of thought commits to its first equation and has to live with it. A tree of thoughts might try 13 - 9 = 4, 10 - 4 = 6 and 4 + 9 = 13 as first steps, ask the model which leftover numbers can still reach 24, drop the dead end (10, 13 and 13 cannot make 24), and end at (10 - 4) * (13 - 9) = 24.

What the paper showed

The authors picked three tasks that GPT-4 found hard even with chain of thought, because each needs planning or search: Game of 24, a creative writing task with fixed ending sentences, and 5x5 mini crosswords. With GPT-4, the tree search solved far more Game of 24 puzzles than chain of thought prompting or than sampling several chains and taking the majority answer, and it also did better on the other two tasks.

The price is calls. Every candidate step and every evaluation is a separate request, so one solved problem can take dozens of model calls where chain of thought costs one. That trade makes sense for hard problems and none for simple ones.

Tree of thought in a single prompt

You can approximate the idea in one message: ask for several distinct approaches, a verdict on each against the facts you gave, and development of only the survivor. The tabs below send the same problem both ways.

Direct answerTree of thought

EditCopy

Our web app is fast for users in Europe but takes about 12 seconds to load for users in Australia. The server is in Frankfurt. When the page loads, the browser makes 40 small API requests, one after another. What is the cause and what should we do?

Try it

Example replyReplies vary between models and runs.

The main cause is the physical distance between Australia and your server in Frankfurt. Every request has to travel to Europe and back, and with 40 requests on each page load, that delay adds up.

The standard fix is a CDN (content delivery network). A CDN keeps copies of your site on servers around the world, so Australian users download from a nearby location instead of from Frankfurt. Most providers let you put one in front of an existing site without changing your code.

You can also compress images and enable browser caching to reduce load time further.

The direct reply is not absurd; it even counts the 40 requests, but it took the first explanation that fit ("distance") and ran with it. The branching reply checked each explanation against the details in the prompt, which is how it noticed that "one after another" matters more than "Australia", and that the CDN would leave the API calls untouched.

A reusable template

The same structure works for design choices and plans. Fill in the problem and the criteria you care about; the reply shows the default filled in.

Branch, judge, choose

EditCopy

Fill inproblemcriteriabranches

Problem: Our team of three needs to choose how to deploy a small Node.js API. Criteria: monthly cost, time to set up, ongoing maintenance Propose 3 genuinely different approaches. For each, rate it against every criterion in one short line and give a verdict: keep or drop. Then develop only the best approach into five concrete first steps. If two approaches tie, say what information would decide between them.

Try it

Example replyReplies vary between models and runs.

Platform as a service (a managed app host) Cost: moderate, rises with traffic. Setup: under an hour from a Git repo. Maintenance: low, the host patches the OS. Keep.

Your own virtual server Cost: lowest. Setup: a day for the server, TLS and a process manager. Maintenance: high, updates and restarts are yours. Drop for a team of three.

Serverless functions Cost: very low at small scale. Setup: moderate, an existing server app may need restructuring. Maintenance: low, but cold starts add latency. Keep as runner up.

Best: platform as a service. First steps:

1. Add a start script and read the port from an environment variable.
2. Move secrets into the host's environment settings.
3. Connect the repository and deploy the main branch.
4. Add a health check endpoint.
5. Set a spending alert.

Where the single prompt falls short

The one-prompt version keeps the vocabulary of the method but loses most of the machinery.

• No real backtracking. The model writes all branches and verdicts in one pass. If step 3 of the chosen branch fails, nothing sends it back to step 1.
• The judge is the author. The same reply that proposed an idea also grades it, so it tends to favour the branch it already had in mind. In the paper the evaluation is a separate call made after the candidates exist.
• Branches are not independent. Ideas listed in one reply influence each other and often end up as variations on one theme. Asking for "genuinely different" approaches, as the template does, pushes against this but does not remove it.
• Reasoning models already branch. Models that think before they answer try and discard approaches internally. For them the prompt adds less accuracy; its remaining value is that you see the options that were rejected and can disagree with the reasoning.

Running a real tree search in code

The full method is a loop in your program: generate candidate steps, score each partial solution in a separate call, keep the best few, repeat. This is a minimal breadth-first version with the OpenAI Python SDK; the same shape works with any provider.

from openai import OpenAI

client = OpenAI()
MODEL = "your-model-id"  # e.g. from your provider's model list

def ask(prompt, temperature=0.7):
    response = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        temperature=temperature,
    )
    return response.choices[0].message.content.strip()

def propose(problem, path, k=3):
    steps = "\n".join(path) or "(none yet)"
    prompt = f"Problem: {problem}\nSteps so far:\n{steps}\nPropose the next step only."
    return [ask(prompt) for _ in range(k)]

def score(problem, path):
    steps = "\n".join(path)
    prompt = (f"Problem: {problem}\nPartial solution:\n{steps}\n"
              "How likely is this to lead to a correct solution? Reply with a number from 1 to 10 only.")
    try:
        return float(ask(prompt, temperature=0))
    except ValueError:
        return 0.0

def tree_of_thought(problem, depth=3, keep=2):
    frontier = [[]]
    for _ in range(depth):
        candidates = [path + [step] for path in frontier for step in propose(problem, path)]
        candidates.sort(key=lambda p: score(problem, p), reverse=True)
        frontier = candidates[:keep]
    return frontier[0]

With depth=3, keep=2 and three proposals per state, one run makes about thirty calls. For many tasks, self-consistency prompting (several full answers, majority vote) or a fixed sequence of steps with prompt chaining gets most of the benefit for fewer calls. Reach for a tree when the task needs search: many possible first moves, and a way to tell a dead end early.