Temperature and Top-p: Controlling How a Model Gambles
Temperature does not make a model more creative, more careful, or more intelligent. It reshapes the probability list before a token is drawn from it. Understanding that one sentence tells you exactly when to change it and when to leave it alone.
Fourth in a series on how language models actually work.
The model proposes; sampling disposes
The model hands back a score for every possible next token. Something then has to pick one. That picking step is entirely separate from the model, it runs on your side of the boundary, and it is where temperature and top-p live.
Temperature scales the scores before they become probabilities. Low temperature exaggerates the gaps, so the leader takes almost everything. High temperature flattens them, so the also-rans get a real chance. The model’s ranking never changes — only how sharply that ranking is enforced.
What the settings actually do
Temperature 0 takes the top token every time. Given identical input this is near-deterministic — though not perfectly so in practice, because floating-point work on GPUs can vary between runs.
Temperature around 0.7 to 1.0 samples roughly in proportion to what the model learned. This is the default for conversation.
Above about 1.2 the tail starts winning often enough to derail sentences. Output becomes not so much creative as incoherent — the model is being forced to pick words it rated as unlikely.
Top-p (nucleus sampling) works differently and more sensibly. It sorts candidates, keeps just enough of them to cover a set share of the probability — 0.9, say — and discards the rest before sampling. When the model is confident, that shortlist may hold two tokens; when it is genuinely unsure, fifty. It adapts, where temperature applies the same pressure regardless.
Top-k keeps a fixed number of candidates. It is blunter than top-p and largely superseded by it.
Choosing a setting
Pick by consequence, not by mood.
- Extraction, classification, structured output, code that must run: temperature 0. There is one right answer; sampling only adds ways to be wrong.
- Conversation, explanation, drafting: 0.7 with top-p around 0.9. Enough variation to avoid stilted repetition, not enough to wander.
- Brainstorming, naming, deliberately varied options: up to about 1.0, and generate several candidates rather than pushing temperature higher. Ten samples at 0.9 beat one sample at 1.5, every time.
Two things worth avoiding. Setting both temperature and top-p aggressively compounds them in ways that are hard to reason about — move one. And raising temperature to fix dull output usually treats the wrong cause: dullness is far more often a weak prompt than an over-confident distribution.
What temperature cannot do
It cannot make a model more accurate. Temperature 0 is not “careful mode” — it is “most likely mode”. If the most likely continuation is wrong, temperature 0 returns that wrong answer with perfect consistency, which is arguably worse than returning it sometimes, because the failure looks stable.
It cannot add knowledge. Sampling only ever chooses among candidates the model already produced. If the right answer is not in that list, no setting will find it.
And it does not control the length or the tone of an answer, though it is often blamed for both. Those come from the prompt.
What to take away
Temperature and top-p sit after the model, shaping how strictly its own ranking is obeyed. Turn them down when there is one correct answer, leave them near default for prose, and reach for multiple samples rather than extreme values when you want variety. Above all, do not treat them as a quality dial — they trade consistency against variety, and nothing else.
Next in the series: the context window, and what the model can genuinely see.