LLM Temperature: What It Does and How to Set It

Temperature is a number that controls how much randomness a language model uses when it picks its next word. At a low temperature the model takes the most likely word almost every time, so answers are focused and repeatable. At a high temperature the chances spread out, so answers vary more and sometimes wander off. Temperature changes how the model chooses among words it already considers; it does not change what the model knows.

How a model picks the next word

A model writes one token at a time, where a token is a word or a piece of a word (see tokens and the context window). At each step it gives every token in its vocabulary a score, called a logit. A function called softmax turns those scores into probabilities: each score is exponentiated and then divided by the sum of all of them, so higher scores get bigger shares and all the shares add up to 1. The model then draws one token at random according to those shares.

Temperature enters just before softmax. Every score is divided by the temperature T:

probability(word) = exp(score / T) / sum of exp(score / T) over all candidates

With T below 1, the gaps between scores grow, so the leader pulls further ahead. With T above 1, the gaps shrink, so the underdogs catch up. At T = 1 you get the model's own probabilities. At T = 0 the division is undefined, so APIs treat it as "always take the top token", which is called greedy decoding.

Try it: the temperature slider

The demo below continues the phrase "My favorite programming language is" with six candidate words. The scores are illustrative, picked for this demo; a real model scores a vocabulary of tens of thousands of tokens or more, and a language name may be split into several tokens. The math is the real one: the bars are the softmax of these scores at the temperature you choose, and the sample button draws ten words from them.

My favorite programming language is …Temperature1.0

Next word probabilities

Python46%

JavaScript28%

Rust14%

C7.6%

Haskell3.4%

COBOL0.8%

Samples

Sample 10 times

Press Sample to let the model pick a word ten times.

At 1.0, Python gets about 46% and COBOL under 1%. Drag down to 0.5 and Python rises to about 67% while COBOL all but disappears. Drag up to 2.0 and Python falls to about 32% while COBOL climbs to about 4%, so about one round of ten samples in three will include it at least once. Notice what never happens: the order of the words stays the same at every setting. Temperature cannot make COBOL more likely than Python; it only widens or narrows the gaps.

A real reply is hundreds of these picks in a row, and each pick becomes part of the context for the next one. One unusual word early on changes everything that follows, which is why a high-temperature reply drifts much further than a single word in this demo suggests.

When to use a low or high temperature

Two cautions. First, a low temperature does not make an answer true. If the model's most likely answer is wrong, temperature 0 gives you that wrong answer every time, just consistently; see AI hallucination. Second, very high values (well above 1, on APIs whose scale goes to 2) often produce text that loses the thread or stops making sense, because unlikely tokens get picked more and more often.

Randomness is sometimes the point. Self-consistency prompting runs the same reasoning question several times at a non-zero temperature on purpose, then takes the answer most runs agree on.

Temperature, top_p and top_k

Two other settings also control sampling, and APIs often expose them next to temperature.

top_p (nucleus sampling) keeps only the most likely tokens whose probabilities add up to p, then samples among those. Using the demo's numbers at temperature 1: Python, JavaScript and Rust add up to about 88%, and adding C passes 90%. So with top_p = 0.9 the model samples only among those four; Haskell and COBOL can never appear. Unlike temperature, top_p adapts: when the model is confident, few tokens make the cut, and when it is unsure, many do.

top_k keeps a fixed number of the most likely tokens, such as the top 40. Some APIs offer it and others do not.

Providers generally recommend changing either temperature or top_p, not both, because the effects stack in ways that are hard to predict.

Setting temperature in the API

The chat apps hide the setting, but the APIs take it as a parameter. The range depends on the provider: OpenAI's Chat Completions API accepts 0 to 2, and Anthropic's Messages API accepts 0 to 1. Some reasoning models accept only their default value.

from openai import OpenAI

client = OpenAI()
MODEL = "your-model-id"  # e.g. from your provider's model list

response = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Write a SQL query that counts orders per customer."}],
    temperature=0.2,
)
print(response.choices[0].message.content)

import anthropic

client = anthropic.Anthropic()
MODEL = "your-model-id"  # e.g. from your provider's model list

message = client.messages.create(
    model=MODEL,
    max_tokens=500,
    temperature=0.2,
    messages=[{"role": "user", "content": "Write a SQL query that counts orders per customer."}],
)
print(message.content[0].text)

Why chat apps hide it, and what to do instead

The ChatGPT, Claude and Gemini apps choose sampling settings for you and show no slider. That keeps the apps simple, and for most requests, rewording the prompt changes the answer far more than a sampling setting would.

So in a chat app, the prompt is your control. Asking for one answer gets you something close to the model's first choice. Asking for many answers that differ from each other gets you variety at any temperature, because the model now has a reason to avoid its first choice. Switch between the tabs and change the product to see the difference.

One answerTen different answers

EditCopy

Fill inproduct

Suggest a name for a habit tracker app.

Try it

Example replyReplies vary between models and runs.

Streakly. It is short, easy to remember, and points to the daily streaks that keep people coming back.

The regenerate button in a chat app samples a new reply from the same prompt, which is a quick way to see how much the answers vary. If they vary more than you want, the fix is usually a more specific prompt: a precise task narrows the range of good answers, and that makes runs more alike. The prompt writing guide covers what to add.