Lensa ML
Lensa ML

Sampling: Temperature, Top-k & Top-p

Step 1 of 6: From Logits to a Choice

Scores for every next token

"The capital of France is ___"logit zsoftmax probabilityParis9.296.2%a5.11.6%the4.61.0%located4.00.5%known3.40.3%not2.90.2%home2.50.1%one2.2<0.1%Lyon1.6<0.1%situated1.2<0.1%Illustrative logits for 10 of the 32,000 vocabulary tokens, softmax over these 10
Top token
"Paris"
Top probability
96.2%
Entropy
0.32 bits

"Paris" has a logit 4.1 higher than the runner-up, and softmax turns that gap into 96.2% of the probability. With an entropy of 0.32 bits, the answer is close to certain.

Switch between Factual prompt and Story prompt and compare the two sets of bars. Softmax computes p_i = exp(z_i) / Σ exp(z_j), so only differences between logits matter: adding 5 to every logit changes nothing. A gap of 4.1 between two logits becomes a probability ratio of e^4.1, about 60 to 1. Entropy, H = −Σ p log₂ p, measures how many bits of uncertainty remain. A real 7B chat model scores all 32,000 vocabulary tokens at every step, and the long tail of tiny probabilities still adds up.

Step 2 of 6: Greedy Decoding

Always take the top token

Greedy output so farOnceuponatime,therewasaNext-token choices after "a" (top 3)little25.6% greedy pickyoung18.9%small12.7%Illustrative next-token table keyed on the last token
Tokens generated
0 / 12
Greedy pick probability
25.6%
Repeated tokens
0

Greedy takes the longest bar, "little" at 25.6%, and drops the other 74.4% of the probability. Press Step to watch it continue.

Press Play, or use Step to advance one token at a time, and follow the text as it grows. Greedy decoding is argmax at every position, the same as temperature 0. It is fast and repeatable, which is why it is common for short factual answers and classification-style prompts. Beam search keeps several candidate sequences instead of one, but it repeats even more on open-ended text. Holtzman et al. (2019) showed that maximizing likelihood produces bland, looping text, and that human writing rarely follows the most probable path.

Step 3 of 6: Temperature

Sharpen or flatten with logits / T

Temperature T1.00
"Once upon a time, there was a ___"p_i = softmax(z_i / 1.00)little25.6%young18.9%small12.7%king10.4%girl8.5%man7.0%beautiful5.7%dragon4.2%princess3.8%boy3.1%at T = 1.00at T = 1 (illustrative logits)
Entropy
3.01 bits
Top-1 probability
25.6%
Effective choices
8.1

T = 1 leaves the model's distribution unchanged: the solid bars match the dashed outlines. The sample spreads over about 8.1 effective choices.

Drag Temperature T from 0.1 to 2.0 and watch the bars move away from the dashed T = 1 outlines. Dividing every logit by T scales all the gaps at once, so the ranking of tokens never changes, only how much the top tokens dominate. As T approaches 0 the result approaches greedy decoding, and as T grows it approaches a uniform distribution. The name comes from the Boltzmann distribution in physics. Effective choices, 2^H, reads entropy as an equivalent number of equally likely tokens.

Step 4 of 6: Top-k

Keep the k most likely tokens

k (tokens kept)3
"Once upon a time, there was a ___"renormalized over the 3 kept tokenslittle44.7%young33.1%small22.2%kingcut (10.4%)girlcut (8.5%)mancut (7.0%)beautifulcut (5.7%)dragoncut (4.2%)princesscut (3.8%)boycut (3.1%)k = 3keptbefore renormalizingcut
Tokens kept
3
Mass removed
42.8%
Top-1 after renorm
44.7%

On the story prompt, k = 3 cuts 42.8% of the probability, including plausible words like "king". A fixed k ignores how spread out the distribution is.

Drag k from 1 to 10 and switch prompts to see how the cut affects each one. Top-k sorts the tokens, keeps the k highest, and divides each kept probability by their total so they sum to 1 again. Fan et al. (2018) used top-k with k = 10 for story generation. The weakness is that k is fixed: on a peaked distribution it lets in tokens the model considers nearly impossible, and on a flat one it removes reasonable words. Many APIs default top_k to off.

Step 5 of 6: Top-p (Nucleus)

Keep the smallest set that covers p

p (cumulative probability)0.90
"Once upon a time, there was a ___"littleyoungsmallkinggirlmanbeautifuldragonprincessboyp = 0.90nucleus: 8in nucleuscumulative probabilitycut
Nucleus size
8
Mass kept
93.0%
Top-1 after renorm
27.5%

The cumulative curve crosses p = 0.90 at token 8, so 8 tokens stay and are renormalized. Switch prompts: the same p keeps 1 on the factual prompt.

Drag p from 0.5 to 1.0 and toggle between the two prompts while watching the nucleus line. Top-p, also called nucleus sampling, was proposed by Holtzman et al. (2019) to fix the fixed-k problem. Sorting the tokens and walking down the cumulative curve until it reaches p means the cut adapts to the shape of each distribution. Values of 0.9 to 0.95 are common. A related filter, min-p, keeps tokens whose probability is at least some fraction of the top token's probability.

Step 6 of 6: Putting It Together

Temperature, filters, then sample

Temperature T0.70
Top-k (10 = off)10
Top-p (1.00 = off)0.90
T 0.7010 tokens›top-k 1010 tokens›top-p 0.907 tokens›sample"small"final probability · draws out of 200little36.0% 74young23.5% 53▸small13.2% 18king10.0% 24girl7.5% 15man5.6% 12beautiful4.2% 4dragoncut 0princesscut 0boycut 0probability after filterscount in 200 seeded draws
seed 1
Sampled token
"small"
Tokens in play
7
Distinct in 200 draws
7
Top-1 probability
36.0%

7 tokens survive the filters and the 200 draws hit 7 of them. Press Resample: the first draw changes with the seed, while the counts stay close to the bar lengths.

Pick a preset, then press Resample several times and compare the sampled token with the counts. A common order, used by Hugging Face Transformers, applies temperature first, then top-k, then top-p, then draws from what is left. Other libraries order the steps differently, so the same numbers can behave slightly differently across tools. Each draw maps a uniform random number onto the cumulative probabilities. Fixing the seed makes a run reproducible, which helps when debugging prompts or comparing settings.

What are temperature, top-k, and top-p in LLM sampling?

Sampling is how a language model turns its scores for the next token into one chosen token. At each step the model outputs a logit for every token in its vocabulary, softmax turns the logits into probabilities, and a decoding rule picks one. Temperature reshapes those probabilities, while top-k and top-p remove unlikely tokens before the draw.

Every chat assistant, code completion tool, and text generator exposes some of these settings, usually as temperature, top_k, and top_p in an API or a config file. Low temperature suits code, extraction, and factual answers where one output is correct. Higher temperature with a top-p filter suits chat and creative writing, where varied wording helps and repetition hurts. The same model can feel precise or inventive depending only on these few numbers.

Key ideas

  • A language model outputs a probability for every vocabulary token, and a decoding rule picks one of them.
  • Greedy decoding always takes the top token, so it is deterministic and can fall into repetition loops.
  • Temperature divides the logits by T: below 1 it sharpens the distribution, above 1 it flattens it.
  • Top-k keeps a fixed number of tokens, which is too many for confident predictions and too few for open ones.
  • Top-p keeps the smallest set of tokens that covers probability p, so its size adapts to each step.
  • Libraries apply these steps in different orders, and a fixed random seed makes sampled output reproducible.

Short takes on this topic

Common questions

Does temperature 0 make an LLM fully deterministic?

In principle temperature 0 means greedy decoding, so the same prompt gives the same output. In practice, hosted models can still vary slightly because of floating-point differences in batched GPU computation. Two tokens with nearly equal logits can swap order between runs.

Should I change temperature and top-p at the same time?

Both control randomness, so changing both makes the effect hard to predict. A common approach is to set top-p once, for example to 0.9, and tune temperature. Some API providers recommend adjusting only one of the two.

What temperature should I use for code generation?

Low values, from 0 to about 0.3, work well when one correct answer is wanted. When generating many candidate programs and testing each, a higher temperature such as 0.8 gives more diverse attempts, which raises the chance that at least one passes.

Why do LLMs repeat themselves?

Once a phrase appears in the context, the model often assigns a high probability to continuing it, and greedy or low-temperature decoding follows that path every time. Sampling with top-p reduces this. Repetition penalties and frequency penalties lower the logits of tokens that already appeared.

Related explainers

References

  1. Holtzman et al. (2019), "The Curious Case of Neural Text Degeneration"
  2. Fan, Lewis, and Dauphin (2018), "Hierarchical Neural Story Generation"
  3. Nguyen et al. (2024), "Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs"

Last updated