LLM Temperature:
What It Does, With Examples

Short answer: LLM temperature is a setting that changes how a language model picks its next token. The model scores every possible next token; temperature divides those scores before they are turned into probabilities. A low temperature sharpens the distribution, so the top choice wins almost every time and output is focused and repetitive. A high temperature flattens it, so less likely tokens get picked more often and output becomes more varied, and also more likely to drift. Temperature does not add knowledge or make answers more accurate.

A large language model writes one token at a time. At every step it produces a score, called a logit, for each token in its vocabulary. A decoding step then turns those scores into probabilities and selects the next token.

Temperature sits in that decoding step. It is a request-level setting: it doesn’t retrain the model, change its weights or give it new information. It only changes how concentrated or spread out the probabilities are when the model samples. If you want to see where those scores come from, the how a transformer works lesson walks through the forward pass.

Two practical consequences:

• The same prompt can produce different answers at the same temperature, because sampling is random by design.
• Every token is sampled with the same temperature, so a small change compounds over a long answer. The longer the output (and the fuller the context window), the more chances there are for a low-probability token to send the text somewhere new.

Logits become probabilities through the softmax function. Temperature T divides every logit before softmax:

In plain text: P(token i) = exp(z_i / T) / Σ_j exp(z_j / T), where z_i is the logit of token i and T > 0.

What happens as T changes:

• T = 1: the logits are used as they are. This is the model’s “native” distribution.
• T < 1: dividing by a number below 1 stretches the gaps between scores, so the top token takes a bigger share.
• T > 1: the gaps shrink, so the distribution becomes flatter and unlikely tokens get more probability.
• T → 0: the top token’s probability approaches 100%. Many implementations treat a temperature of 0 as “always pick the most likely token” (greedy decoding), because dividing by zero is undefined.

Temperature never changes the ranking of tokens. The most likely token stays the most likely; only the size of its lead changes. If two tokens have equal scores, they stay equally likely at any temperature.

This is a simplified view. Real serving stacks may combine temperature with other rules (top-p, top-k, penalties), and some APIs restrict or ignore the parameter for certain models.

The numbers below are illustrative. They are made-up logits for three candidate next words after the prompt “The sky is”, not output from a real model. A real model scores tens of thousands of tokens at every step.

How to read it:

• At T = 0.2, “blue” is picked about 99 times in 100. Run the prompt ten times and you’ll almost always get the same sentence.
• At T = 1.0, “blue” still leads, but roughly one run in three picks something else.
• At T = 1.5, the odd choice (“falling”) shows up about one time in five. That might be a creative spark in a poem and a derailment in a support answer.

You can check any column yourself: divide each logit by T, take exp() of each result, then divide each by their sum.

A toy next-token distribution

At lower temperatures, the leading candidate dominates more of this toy distribution. Raising the temperature gives the alternatives more room. Sampling still means a token is selected from the distribution; a higher probability is not a guarantee that a particular token will be chosen.

Top-p sampling, also called nucleus sampling, takes a different approach. Instead of reshaping the probabilities, it cuts off the tail. The model sorts tokens from most to least likely, keeps the smallest set whose probabilities add up to at least p, renormalizes that set, and samples only from it.

Using the T = 1.0 column above with top_p = 0.8: “blue” (62.9%) alone isn’t enough, so “clear” is added (62.9 + 23.1 = 86% ≥ 80%). “falling” is dropped entirely. The model then samples between “blue” and “clear” only.

Why it’s useful: the size of the kept set adapts. When the model is confident, the nucleus may be one or two tokens; when it’s unsure, many tokens fit inside the same p. OpenAI’s API reference describes top_p as “an alternative to sampling with temperature”, with 0.1 meaning only the tokens in the top 10% of probability mass are considered.

Top-k is the blunter cousin: keep the k most likely tokens, whatever their probabilities, and sample among them.

The controls can be combined: for example, Google’s Gemini API documentation says Gemini models use top-p (nucleus) sampling or a combination of top-k and nucleus sampling. OpenAI’s reference says it generally recommends altering temperature or top_p, but not both. Changing one control at a time also makes your experiments easier to interpret.

Numeric scales are not universal. A “0.7” on one API is not necessarily the same as “0.7” on another. From the vendor references we checked on 2026-10-10:

Always check the page for the exact model and endpoint you call. Some newer models restrict or drop the parameter.

There is no single “best” temperature. The ranges below are common starting points on a 0–1-style scale, not rules. Test them on your own prompts and measure the results.

If you build retrieval-augmented generation, grounding comes from the retrieved documents and the prompt. Lowering temperature only makes the model’s choice among tokens more predictable.

A simple way to tune it

1. Confirm that your model and endpoint accept temperature, and note the documented range.
2. Decide what “good” means for the task (valid schema, matches source, varied ideas).
3. Fix the prompt, model version and other settings; change only temperature (or only top-p).
4. Run several samples per setting on representative inputs and score them against your rubric.
5. Record the full configuration: model ID, prompt version, temperature, top-p and test date.

• It doesn’t make answers true. OpenAI’s prompt-engineering guidance separates temperature from truthfulness: higher values make output more random, not more accurate. A focused answer can still be wrong.
• It doesn’t guarantee repeatability. Even at 0, Anthropic’s docs note results are not fully deterministic. Model updates and serving differences can change output too.
• It doesn’t change what the model knows or how much text it can read at once. That’s the context window.
• It isn’t a safety filter. Low temperature won’t stop a model from following injected instructions. See indirect prompt injection.

• OpenAI API reference, Responses create (temperature, top_p): https://developers.openai.com/api/reference/cli/resources/responses/methods/create
• OpenAI Help Center, best practices for prompt engineering: https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-the-openai-api
• Anthropic Messages API reference: https://docs.claude.com/en/api/messages/create
• Google Gemini API, GenerationConfig: https://ai.google.dev/api/generate-content
• Hugging Face Transformers, text generation strategies: https://huggingface.co/docs/transformers/main_classes/text_generation

Vendor references checked on 2026-10-10. Parameter behavior is model- and interface-specific; check the page for the exact model and endpoint you call.