Temperature vs top-p: reshape every bar or cut the tail

A language model writes one token at a time, where a token is a word or part of a word. For each step it gives every candidate token a probability, then draws one at random. Temperature and top-p both change those probabilities before the draw, in different ways.

1. 1
   The bars come from the explainer's story prompt, Once upon a time, there was a ___, with ten candidate tokens. The model gives each a raw score called a logit, from 4.4 for little down to 2.3 for boy. Softmax turns logits into probabilities that add up to 1: it raises e to each logit and divides by the total. Temperature T is a number that divides every logit before softmax. At T = 1 nothing changes, so little gets 25.6% and boy 3.1%.
2. 2
   At T = 0.5 the gaps between logits double, so little rises to 44.2% and boy falls to 0.7%. Every bar moves, and none drops to zero. The ranking never changes. A T above 1 does the opposite and flattens the bars toward equal.
3. 3
   Top-p, also called nucleus sampling, leaves the shape alone and cuts the tail. It sorts tokens from most to least likely and keeps adding them until their total reaches p. With p = 0.9, the first seven tokens cover 88.8% and the eighth, dragon, takes the total to 93.0%, so eight stay. Princess and boy are cut, and the kept bars are divided by 0.930 so they add up to 1 again: little becomes 27.5%. On the explainer's factual prompt, Paris alone has 96.2%, so the same p keeps one token.

More bites

In 1 picture

Temperature in 1 picture: from confident to spread out

2 min read