Continuous vs static batching: why LLM servers refill slots every step
An LLM server generates one token per step for every request in its batch, because one read of the model's weights can serve many sequences at once. Requests want different numbers of tokens, so the way the server fills its batch decides how much of the GPU sits idle and how long queued requests wait.
2 min read · Lensa ML
- Eight requests wait in a queue at step 0, needing 3, 12, 5, 4, 10, 2, 6 and 8 output tokens. The server has 4 batch slots. Each block is one request, R1 to R8, and its width is the number of steps it takes. Steps are counted from 0 on the axis.
- Static batching admits R1 to R4 and runs until the longest, R2 with 12 tokens, is done. R1 finishes after 3 steps, and its slot sits idle for the next 9, the hatched area. R5 to R8 wait 12 steps before producing their first token. The whole queue takes 22 steps, and the slots are busy only 57% of the time.
- Continuous batching, also called iteration-level scheduling, checks the slots at every step. R1 finishes after 3 steps and R5 takes its slot on the very next one; R6, R7 and R8 follow as R4, R3 and R6 finish. R5 to R8 start at steps 3 to 6 instead of 12, the queue finishes in 14 steps, and the slots are busy 89% of the time. A request finishes after 8.5 steps on average, against 12.25.
Takeaway
Continuous batching is the default in serving systems such as vLLM, TensorRT-LLM and Hugging Face TGI. It raises throughput and cuts waiting without changing the model's output. In practice the batch is limited by KV cache memory rather than a fixed slot count, which is why it pairs with a paged KV cache.
Go deeper · interactive
Continuous Batching
On step 3, press Play to run both schedulers on the same requests, then change the number of batch slots.
Open the explainer