Lensa ML
Lensa ML

Continuous Batching

Step 1 of 6: Why Batch at All

One weight read serves every sequence

Batch size (sequences decoding together)8
Output tokens/s (illustrative GPU: 300 TFLOP/s, 2 TB/s)02k4k6k8k116324864batch size1,024 tok/sperfectly linear7B chat model, fp16, 512 cached tokens per sequence; illustrative timings
Throughput
1,024 tok/s
Time per decode step
7.8 ms
Speedup vs batch 1
7.0×
Bound by
memory

8 sequences share one read of the weights, so throughput is 7.0× batch 1 while each step slows from 6.9 ms to 7.8 ms. Weights are still 86% of the bytes read.

Drag Batch size from 1 to 64 and compare the curve with the dashed line. The step time here is max(FLOPs ÷ 300 TFLOP/s, bytes ÷ 2 TB/s), and bytes are the 13.5 GB of weights plus each sequence's own cache: 512 tokens × 0.5 MiB, about 0.27 GB per sequence. Weights are shared across the batch, but caches are not, so the curve stays below linear. Real servers see the same shape, and it is why per-user speed and total throughput are quoted separately.

Step 2 of 6: Static Batching

Everyone waits for the longest request

Requests in the batch6
One row per request, one cell per decode step (seeded lengths)04812162024R111R210R319R421R513R68batch ends at step 21generating a tokenfinished, slot idle
Steps the batch runs
21
Useful slot-steps
82
Idle slot-steps
44
Utilization
65%

R4 needs 21 steps, so every other request holds its slot until then. 44 of 126 slot-steps (35%) compute nothing, and queued requests wait outside.

Move Requests in the batch and press New request lengths to draw another set. Utilization equals the sum of the lengths divided by n × the longest length, so one long answer drags it down no matter how many short ones it shares a batch with. Output length is unknown until a sequence emits its end token, so a static scheduler cannot sort requests by length ahead of time. Early serving stacks that batched whole requests together had exactly this problem.

Step 3 of 6: Iteration-Level Scheduling

Join and leave the batch at every step

Batch slots4
Arrivals (same 14 seeded requests for both schedulers)Static batching0 done, 0% busy1234Continuous batching0 done, 0% busy1234request decodingslot held, idleempty or queued
Decode step
0 / 32
Tokens, static
0
Tokens, continuous
0
Finished, static vs cont.
0 vs 0

Both schedulers get the same 14 requests on 4 slots. Press Play or Step to run one decode step at a time.

Press Play or Step, then change Batch slots and replay. Orca (Yu et al., OSDI 2022) named this iteration-level scheduling: the scheduler decides batch membership before every forward pass instead of once per request. Orca also used selective batching. Linear layers process every token in the batch as one matrix, while attention runs per sequence, since each one has a different cache length. The timeline leaves out prefill, which a newly admitted request needs before its first token.

Step 4 of 6: Memory Sets the Batch

The KV cache runs out before compute does

Context length per sequence (tokens)4,096
Batch size (sequences)16
Memory on one 80 GB GPU13.5 GB + 16 × 4,096 × 512 KiB = 47.8 GB80 GBweights 13.5 GBKV cache 34.4 GBBatch size064128192256memory cap 30compute-bound from ~150fp16 weights and cache; activations and runtime buffers not counted
KV per sequence
2.1 GB
Max batch that fits
30
Batch for compute-bound
~150
Memory used
47.8 GB

At 4,096 tokens, the 66.5 GB left after the weights holds 30 sequences, well short of the ~150 it would take to keep the compute units busy. Cache memory caps the batch here.

Drag Context length per sequence and Batch size and watch the memory cap move. The cap is (80 GB − 13.5 GB) ÷ (context × 0.5 MiB): at 4,096 tokens each sequence needs 2.1 GB, so 30 fit. The ~150 marker comes from the GPU's ridge point. Decode does about 2 FLOPs per parameter per token and reads 2 bytes per parameter in fp16, so it needs about 150 sequences to reach 150 FLOPs per byte. The marker ignores cache reads, which push the real threshold higher.

Step 5 of 6: Paged KV Cache

Allocate fixed-size blocks on demand

Reserved length per request (tokens)4,096
Block size (tokens)16
8 requests, seeded final lengths (prompt + output)top bar: reserve the max lengthbottom bar: 16-token blocksR1256R2209R31281R4660R5695R6487R71052R81891tokens storedallocated, unusedtokens (reserved)
Waste, max reservation
13.8 GB
Waste, 16-token blocks
40 MB
Requests fit, reserved
30
Requests fit, paged
153

Reserving 4,096 tokens per request leaves 80% of the reserved memory empty. With 16-token blocks only the last block of each request has spare room, 40 MB in total, so about 153 requests of this mix fit instead of 30.

Drag Reserved length per request and Block size, then compare the two bars in each row. PagedAttention (Kwon et al., 2023) borrows virtual memory from operating systems: a per-request block table maps logical blocks to physical blocks anywhere in GPU memory, so no contiguous region is needed. The vLLM paper measured that earlier systems used only 20.4% to 38.2% of their KV cache memory for actual token states. Its default block size is 16 tokens, and waste is at most one partly filled block per request.

Step 6 of 6: Throughput vs Latency

Time to first token, time per token, chunked prefill

Arrival rate (requests/s)10
Max batch size32
Chunked prefill
Longest forward pass in each time slice (ms)04080120160mean time per output tokensimulated time: 60 requests over 8.5 s, bars capped at 160 msnormal passstall (> 3× mean)
Throughput
980 tok/s
Mean time to first token
360 ms
Mean time per output token
22 ms
Longest pass
291 ms

Each new prompt is prefilled in one pass, and every decoding sequence waits for it: the longest pass takes 291 ms. Turn on chunked prefill to spread that work across passes.

Move Arrival rate and Max batch size, and toggle Chunked prefill. The simulation sends 60 seeded requests with prompts of 256 to 2,048 tokens and outputs of 32 to 256 tokens through the same illustrative GPU model. Time to first token covers queueing plus prefill. Time per output token is the gap between later tokens. Sarathi-Serve (Agrawal et al., 2024) proposed chunked prefill: long prompts are split into pieces that share a pass with ongoing decodes, so they no longer stall them.

What is continuous batching in LLM serving?

Continuous batching is a scheduling method for LLM inference that adds new requests to a running batch and removes finished ones at every decode step, instead of waiting for a whole batch to finish. A server decodes many sequences together because one read of the model weights then serves all of them. With continuous batching, a slot freed by a short answer is refilled on the next step, so the GPU rarely sits idle while a long answer finishes.

Every major LLM serving engine uses it, including vLLM, Hugging Face Text Generation Inference, TensorRT-LLM (where it is called in-flight batching), and SGLang. It usually comes paired with a paged KV cache, because the number of sequences a GPU can run at once is limited by cache memory. Together these two ideas raise serving throughput several times over request-level batching, and they shape the latency figures users see: time to first token and time per output token.

Key ideas

  • Decoding is memory-bound, so batching sequences shares one read of the weights and raises throughput.
  • Static batching holds every slot until the longest request in the batch finishes, which leaves slots idle.
  • Continuous batching admits and retires requests at every decode step, keeping slots full.
  • The KV cache, not compute, usually limits how many sequences fit in a batch.
  • PagedAttention stores the cache in fixed-size blocks, cutting waste to at most one partial block per request.
  • Larger batches raise throughput but slow each token, and chunked prefill limits the stalls new prompts cause.

Common questions

How is continuous batching different from dynamic batching?

Dynamic batching collects requests that arrive within a short time window and runs them as one batch from start to finish. It works well for fixed-size work such as image classification. Continuous batching changes the batch at every token step, which suits generation, where output lengths vary widely and are unknown in advance.

Does continuous batching change the model's output?

No. Each sequence attends only to its own tokens, so other requests in the batch do not affect its logits. Small numerical differences can appear because kernels sum in a different order for different batch shapes. With greedy decoding this can occasionally flip a near-tie between two tokens.

What happens when the KV cache fills up?

The scheduler stops admitting new requests, and if running sequences still need more blocks, it preempts some of them. vLLM either swaps the preempted blocks to CPU memory or frees them and recomputes the cache later from the tokens already generated. Preempted requests resume once memory frees up.

Why do serving benchmarks report both throughput and latency?

A server can raise total tokens per second by packing more sequences into each pass, while each user sees slower tokens and a longer wait for the first one. Throughput measures cost per token, and latency measures the experience of one user. Serving limits are usually set as a latency target, with throughput maximized under it.

Related explainers

References

  1. Yu et al. (2022), "Orca: A Distributed Serving System for Transformer-Based Generative Models", OSDI '22
  2. Kwon et al. (2023), "Efficient Memory Management for Large Language Model Serving with PagedAttention"
  3. Agrawal et al. (2024), "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve"
  4. Pope et al. (2022), "Efficiently Scaling Transformer Inference"

Last updated