Lensa ML
Lensa ML

Speculative Decoding

Step 1 of 6: The Decode Bottleneck

Every token streams all the weights

Tokens to generate32
GPU memory (HBM)weights 13.5 GBCompute units1 token per passevery passTimeline, 32 passesreading weightsmathCompute units busy0.6%Times assume an illustrative GPU: 2 TB/s memory, 312 TFLOP/s fp16
Target passes
32
Weight bytes read
431.4 GB
Decode time
215.7 ms
Compute busy
0.6%

32 tokens need 32 sequential passes and 431.4 GB of weight reads. Each thin sliver is the math; the grey is the GPU waiting on memory.

Drag Tokens to generate from 1 to 256 and watch bytes and time grow in lockstep. The rule behind it is arithmetic intensity. A decode pass does about 2 FLOPs per parameter and reads 2 bytes per parameter in fp16, roughly 1 FLOP per byte. An A100-class GPU needs about 150 FLOPs per byte to keep its math units full. Time per token is therefore set by bandwidth: 13.5 GB at 2 TB/s is about 6.7 ms, a ceiling near 150 tokens per second for one sequence.

Step 2 of 6: A Cheap Draft

A small model guesses γ tokens ahead

Draft length γ4
"The capital of France is"1B draft model guesses 4 tokens, one cheap pass eachd1Parisq 0.82d2.q 0.71d3Itq 0.55d4isq 0.78draft tokens and probabilities q are illustrativeWeight bytes read for 4 tokens7B, one by one53.9 GB1B draft8.0 GB
Draft passes
4
Draft bytes read
8.0 GB
Cost ratio c
0.148
Cost in 7B passes
0.59

4 draft passes read 8.0 GB, about 0.59 of one 7B pass. None of these guesses is trusted yet; the target still has to check them.

Drag Draft length γ from 1 to 8 and compare the two bars. The cost ratio c is the time of one draft pass divided by one target pass. When both models are memory-bound it is close to the ratio of their weight sizes, here 2.0 GB over 13.5 GB, or 0.148. The draft must share the target's tokenizer, since verification compares token IDs. A common choice is a small model from the same family, such as a 1B model drafting for a 7B or 70B model.

Step 3 of 6: Verify in One Pass

Keep the matching prefix plus one token

Draft length γ5
Draft guessesa.Itisknown7B target, one parallel pass over prompt + 5 draftsParis.ItisknownforbonusOutput this roundaccepted drafttoken from the targetrejected (illustrative, seeded)
Drafted
5
Accepted
0
Tokens this round
…
7B passes
0

The target reads the prompt plus all 5 drafts at once and predicts the next token at every position. Press Step to compare each draft with the target's prediction, left to right.

Press Step to check the drafts one at a time, press New draft to reseed the guesses, and drag Draft length γ to change how many get checked. Verification works because a transformer scores every input position in parallel, the same way prefill handles a prompt. The causal mask makes the prediction at position i depend only on the prompt and the drafts before it. KV cache entries written for rejected drafts are dropped before the next round. This greedy check reproduces greedy decoding with the target alone, token for token.

Step 4 of 6: Acceptance and Speedup

How α, γ and draft cost set the gain

Acceptance rate α0.75
Draft length γ4
Draft cost ratio c0.15
0×1×2×3×12345678910draft length γbreak-even1.91×speeduptokens per pass (free draft)
Tokens per 7B pass
3.05
Speedup
1.91×
Best γ
4 (1.91×)
Draft cost γc
0.60

γ = 4 is the best length here. Each round yields 3.05 tokens and costs 1.60 7B passes, a 1.91× speedup.

Drag Acceptance rate α, Draft length γ, and Draft cost ratio c, and follow the amber ring that marks the best γ. The formula assumes each draft is accepted independently with probability α, so a round yields 1 + α + α² + ... + α^γ tokens. At α = 0.75 and γ = 4 that sum is 3.05, and with c = 0.15 a round costs 1.6 target passes, about 1.91 times faster. Real acceptance varies with the text: code and boilerplate are easy to guess, open-ended prose is harder.

Step 5 of 6: Exactly the Same Output

Accept with min(1, p/q), else resample

Draft mismatch0.60
0.00.20.40.6littleqoutyoungqoutsmallqoutkingqoutaccepted draftresampledtarget psimulated (illustrative p, q)
Acceptance rate
73.2%
Resampled share
26.8%
Max gap, draft q vs p
0.268
Max gap, output vs p
0.011

The draft over-weights "king" and is off by up to 0.27, yet the output bars stop at p exactly. Only the cost changes: 26.8% of rounds end in a resample.

Drag Draft mismatch from 0 to 1 and press Resample to rerun the 4,000-sample simulation. The proof takes one line per token. The chance of emitting x is q(x) · min(1, p(x)/q(x)) + (1 − β) · r(x), where r is the normalized residual max(0, p − q). The first term equals min(p, q) and the second equals max(0, p − q), so they sum to p(x). The acceptance rate β = Σ min(p, q) is one minus the total variation distance between p and q.

Step 6 of 6: In Practice

Draft heads, prompt lookup, and batch size

Method
Batch size1
0×1×2×3×4×1248163264128256batch size (log scale), γ = 41.91×Illustrative α and c per method; GPU as in step 1, KV cache reads ignored
Speedup
1.91×
Tokens per pass
3.05
Tokens per verify pass
5
Verify pass
memory-bound

A separate small model with the same tokenizer writes the drafts. At batch 1 the verify pass of 5 tokens still takes one weight read, so checking drafts is nearly free: 1.91×.

Switch Method and drag Batch size from 1 to 256. Medusa (Cai et al., 2024) and EAGLE (Li et al., 2024) verify a tree of candidate continuations in one pass with a tree-shaped attention mask, which lengthens the accepted run beyond a single chain. Prompt lookup decoding needs no extra model at all, which suits summarization, retrieval answers, and code edits where the output copies the input. Published single-sequence speedups are typically 2 to 3 times, and the exact figure depends on the model pair and the task.

What is speculative decoding in LLMs?

Speculative decoding is a way to speed up text generation in which a small, fast model drafts several tokens ahead and the large model checks them all in one forward pass. The large model keeps every draft token that matches what it would have produced, replaces the first one that does not, and the cycle repeats. With the right acceptance rule the output follows the large model's distribution exactly, so speed improves without any change in quality.

Generating one token at a time is limited by memory bandwidth, not arithmetic: each step streams all the model's weights from GPU memory to produce a single token. Speculative decoding spends the idle compute on checking several guesses per weight read. It cuts per-token latency for chat assistants, coding tools, and any single-user setting, and it is built into serving systems such as vLLM, TensorRT-LLM, and llama.cpp. Its benefit shrinks at large batch sizes, where the GPU is already busy.

Key ideas

  • Decoding one token per pass is memory-bound, so the GPU's compute units sit mostly idle.
  • A cheap drafter proposes γ tokens, and the target model checks all of them in one parallel pass.
  • Each round keeps the longest accepted prefix plus one token from the target, so it always makes progress.
  • With acceptance rate α, a round yields (1 − α^(γ+1)) / (1 − α) tokens on average.
  • The accept-with-min(1, p/q) rule and residual resampling keep the output distribution identical to the target's.
  • The speedup is largest for single requests and shrinks as batch size makes decoding compute-bound.

Common questions

Does speculative decoding reduce total compute?

No. It usually increases the total floating-point work, since the draft model runs extra passes and the target scores drafts that end up rejected. It reduces the number of sequential target passes, trading spare compute for lower latency. On a busy server with no spare compute, that trade stops paying off.

Does speculative decoding work with temperature and top-p sampling?

Yes. The acceptance rule works for any pair of distributions, so temperature, top-k, or top-p is applied to both the target's p and the draft's q before the check. Greedy decoding is the special case where both distributions put all their mass on one token. Acceptance rates tend to fall at higher temperatures, because sampled tokens are harder to predict.

How do I choose a draft model?

Pick a model that is much cheaper than the target, often 10 to 20 times smaller. A model from the same family trained on similar data agrees with the target more often, which raises the acceptance rate. Some teams also distill the draft from the target to raise it further.

What is the difference between speculative decoding and speculative sampling?

They are two names for the same idea, published independently around the same time. Leviathan et al. (2022) called it speculative decoding and Chen et al. (2023) called it speculative sampling. Both borrow the term from speculative execution in CPUs, where a processor runs ahead on a guess and discards the work if the guess was wrong.

Related explainers

References

  1. Leviathan, Kalman & Matias (2022), "Fast Inference from Transformers via Speculative Decoding"
  2. Chen et al. (2023), "Accelerating Large Language Model Decoding with Speculative Sampling"
  3. Cai et al. (2024), "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads"
  4. Li et al. (2024), "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty"

Last updated