Builds onKV Cache Quantization Continuous Batching Decoder-Only Transformers

Lensa ML

Serve an LLM

Step 1 of 6: The Brief

Traffic, prompt sizes, and latency targets

GPUs

3

First token

45 ms

Per token

32 ms

Per 1M out

$0.556

Design for 10 req/s · 1,000-token prompts · 300-token answers

Target requests per second10

Prompt length (tokens)1,000

Answer length (tokens)300

Time-to-first-token target1.0 s

Per-token target50 ms

Tokens per request

1,300

Answer tokens/s needed

3,000

Reading speed at target

20 tok/s

The fleet has to produce 3,000 answer tokens every second, while each user sees their first token within 1.0 s and then one every 50 ms. The next steps find out what that costs.

Assumptions

Efficiency vs the ideal roofline100%

Burst headroom1.30×

Model	7B, fp16	Llama 2 7B shape: 32 layers, 32 KV heads of 128 dims
GPU	80 GB · 2 TB/s · 300 TFLOP/s	Illustrative, the same accelerator as the KV cache and continuous batching explainers
Pass time	max(compute, memory)	FLOPs ÷ peak or bytes ÷ bandwidth, whichever is slower
Memory reserve	10%	Activations, fragmentation, runtime buffers
Price	$2.00/GPU-hour	An example; set it in the last step
Not modelled	queueing, prefill stalls	Time to first token assumes an empty queue; headroom stands in for both

Move the five sliders to set the brief. Answer tokens per second is requests per second times answer length, and it is the number the whole fleet must sustain. The per-token target of 50 ms is about 20 tokens a second. That is faster than most people read, which is why chat services aim near it rather than far below it.

Step 2 of 6: Does It Fit on One GPU?

Weights once, a KV cache per request

GPUs

3

First token

45 ms

Per token

32 ms

Per 1M out

$0.556

Design for 10 req/s · 1,000-token prompts · 300-token answers

Weights

KV cache

Requests running at once32

Weights

13.5 GB

KV per request

0.68 GB

Most that fit

85

The KV cache for 32 requests (21.8 GB) is already bigger than the weights (13.5 GB). Beyond a few requests the cache, not the model, decides how many users one GPU can serve: 85 here.

Assumptions

Efficiency vs the ideal roofline100%

Burst headroom1.30×

Model	7B, fp16	Llama 2 7B shape: 32 layers, 32 KV heads of 128 dims
GPU	80 GB · 2 TB/s · 300 TFLOP/s	Illustrative, the same accelerator as the KV cache and continuous batching explainers
Pass time	max(compute, memory)	FLOPs ÷ peak or bytes ÷ bandwidth, whichever is slower
Memory reserve	10%	Activations, fragmentation, runtime buffers
Price	$2.00/GPU-hour	An example; set it in the last step
Not modelled	queueing, prefill stalls	Time to first token assumes an empty queue; headroom stands in for both

Switch Weights between fp16, int8, and int4, then drag Requests running at once past the 80 GB line. Weights cost 6.74 billion parameters times bytes per parameter. The key-value (KV) cache costs 2 (keys and values) × 32 layers × 32 KV heads × 128 dimensions × 2 bytes = 0.5 MiB per token in fp16, for every token of every request. 10% of memory is held back for activations and fragmentation.

Step 3 of 6: One Request's Timeline

Compute-bound prefill, memory-bound decode

GPUs

3

First token

45 ms

Per token

32 ms

Per 1M out

$0.556

Design for 10 req/s · 1,000-token prompts · 300-token answers

Prompt length (tokens)1,000

First token

45 ms

Per token (alone)

7.0 ms

Prefill is

compute-bound

Prefill pushes all 1,000 prompt tokens through in one pass: 13.5 TFLOPs, compute-bound at 45 ms. Each answer token is then a separate pass that re-reads the 13.5 GB of weights for one token of work, so decode is memory-bound and takes 7.0 ms per token with one user.

Assumptions

Efficiency vs the ideal roofline100%

Burst headroom1.30×

Model	7B, fp16	Llama 2 7B shape: 32 layers, 32 KV heads of 128 dims
GPU	80 GB · 2 TB/s · 300 TFLOP/s	Illustrative, the same accelerator as the KV cache and continuous batching explainers
Pass time	max(compute, memory)	FLOPs ÷ peak or bytes ÷ bandwidth, whichever is slower
Memory reserve	10%	Activations, fragmentation, runtime buffers
Price	$2.00/GPU-hour	An example; set it in the last step
Not modelled	queueing, prefill stalls	Time to first token assumes an empty queue; headroom stands in for both

Drag Prompt length and watch the prefill bar grow while the decode ticks stay the same width. Prefill processes all prompt tokens in one pass, about 2 FLOPs per parameter per token, so it is limited by compute. Each decode pass handles one token but still reads all the weights, so it is limited by memory bandwidth: 13.5 GB ÷ 2 TB/s ≈ 6.7 ms before any cache reads.

Step 4 of 6: Batching

Share each weight read across many users

GPUs

3

First token

45 ms

Per token

32 ms

Per 1M out

$0.556

Design for 10 req/s · 1,000-token prompts · 300-token answers

Batch size32

Per token

16 ms

Tokens/s per GPU

1,953

Requests/s per GPU

5.04

32 requests share each read of the weights, so the GPU produces 1,953 tokens a second while each user still gets one every 16 ms. The cost is each request's own cache, read on every step, which is why the time per token slowly rises.

Assumptions

Efficiency vs the ideal roofline100%

Burst headroom1.30×

Model	7B, fp16	Llama 2 7B shape: 32 layers, 32 KV heads of 128 dims
GPU	80 GB · 2 TB/s · 300 TFLOP/s	Illustrative, the same accelerator as the KV cache and continuous batching explainers
Pass time	max(compute, memory)	FLOPs ÷ peak or bytes ÷ bandwidth, whichever is slower
Memory reserve	10%	Activations, fragmentation, runtime buffers
Price	$2.00/GPU-hour	An example; set it in the last step
Not modelled	queueing, prefill stalls	Time to first token assumes an empty queue; headroom stands in for both

Drag Batch size from 1 to the memory limit. The time per token is max(2 × params × batch ÷ 300 TFLOP/s, (weights + batch × context × 0.5 MiB) ÷ 2 TB/s). The weights term is shared by the batch, but each request's cache is not, so the time per token keeps rising. The plan runs the largest batch whose time per token stays under the target.

Step 5 of 6: Scale Out

Little's law and the levers that shrink the fleet

GPUs

3

First token

45 ms

Per token

32 ms

Per 1M out

$0.556

Design for 10 req/s · 1,000-token prompts · 300-token answers

Target requests per second10

in flight = 10 req/s × 9.75 s = 98

GPUs

3

Requests in flight

98

Req/s per GPU

6.28

Each decode step spends most of its time reading caches, not weights. That's why GQA, which shrinks the cache four times, cuts the fleet far more than int4 weights do. Little's law ties it together: 10 req/s that each take 9.75 s means 98 requests are in flight at any moment.

Assumptions

Efficiency vs the ideal roofline100%

Burst headroom1.30×

Model	7B, fp16	Llama 2 7B shape: 32 layers, 32 KV heads of 128 dims
GPU	80 GB · 2 TB/s · 300 TFLOP/s	Illustrative, the same accelerator as the KV cache and continuous batching explainers
Pass time	max(compute, memory)	FLOPs ÷ peak or bytes ÷ bandwidth, whichever is slower
Memory reserve	10%	Activations, fragmentation, runtime buffers
Price	$2.00/GPU-hour	An example; set it in the last step
Not modelled	queueing, prefill stalls	Time to first token assumes an empty queue; headroom stands in for both

Move Target requests per second and toggle the four levers. Requests per GPU is 1 ÷ (answer tokens × time per token ÷ batch + prefill time), and the fleet is the target rate times 1.3 headroom divided by that. Little's law (L = λW) says the requests in flight equal the arrival rate times the time each one spends in the system. Grouped-query attention (GQA) lets several query heads share one key-value head. Mistral 7B and Llama 2 70B both use 8 KV heads; with 8 instead of 32, this model's cache shrinks four times.

Step 6 of 6: Design Review

The design against the brief, and what to change next

GPUs

3

First token

45 ms

Per token

32 ms

Per 1M out

$0.556

Design for 10 req/s · 1,000-token prompts · 300-token answers

✓First token ≤ 1.0 s45 ms

✓Per token ≤ 50 ms32 ms

·GPUs 10 req/s × 1.3 headroom3

·Per 1M answer tokens at $2.00/GPU-hour$0.556

GPU price ($ per hour)$2.00

Fleet per hour

$6.00

Per 1,000 requests

$0.167

Bottleneck

KV reads

Both latency targets are met with 3 GPUs at $0.556 per million answer tokens. Each decode step is limited by reading KV caches, so the best next change is GQA (8 KV heads), saving 2 GPUs.

Assumptions

Efficiency vs the ideal roofline100%

Burst headroom1.30×

Model	7B, fp16	Llama 2 7B shape: 32 layers, 32 KV heads of 128 dims
GPU	80 GB · 2 TB/s · 300 TFLOP/s	Illustrative, the same accelerator as the KV cache and continuous batching explainers
Pass time	max(compute, memory)	FLOPs ÷ peak or bytes ÷ bandwidth, whichever is slower
Memory reserve	10%	Activations, fragmentation, runtime buffers
Price	$2.00/GPU-hour	An example; set it in the last step
Not modelled	queueing, prefill stalls	Time to first token assumes an empty queue; headroom stands in for both

Set GPU price to your provider's rate. Cost per million answer tokens is GPUs × price per hour ÷ answer tokens per hour × 10⁶. The lever ranking re-runs the whole plan with each change switched on and keeps the ones that save a GPU. Open Assumptions and lower Efficiency to see how much the fleet depends on reaching the ideal roofline.

How many GPUs does it take to serve an LLM?

Serving a large language model (LLM) means answering many users at once within latency targets: how long until the first token of an answer appears (time to first token), and how long each later token takes (time per output token). The number of GPUs needed follows from three things: how much memory each request needs, how long each forward pass takes, and how many requests arrive per second.

This case study plans a deployment for a 7-billion-parameter chat model from first principles. Every number is computed from the model's shape and an illustrative 80 GB GPU with 2 TB/s of memory bandwidth and 300 TFLOP/s of compute, the same accelerator used in the KV cache and continuous batching explainers. Each forward pass takes whichever is slower: the arithmetic, or reading its bytes from memory. This is the roofline model.

Key ideas

Weights are loaded once per GPU, but every request keeps its own KV cache, so cache memory decides how many users one GPU can serve.
Prefill is compute-bound and sets time to first token; decode is memory-bound and sets time per output token.
Batching shares each read of the weights across many requests, raising throughput while each user's time per token rises slowly.
The best batch is the largest one that fits in memory and keeps time per token under the target.
Little's law turns a request rate and a latency into the number of requests in flight, which the fleet has to hold at once.
When cache reads dominate a decode step, shrinking the cache (GQA, an int8 cache, shorter contexts) saves more GPUs than shrinking the weights.

Common questions

Why does a 7B model need more than one GPU if it fits in 13.5 GB?

The weights fit easily, but serving many users at once is limited by their KV caches and by time. At 1,000-token prompts and 300-token answers each request needs about 0.68 GB of cache, so about 85 fit beside the weights on an 80 GB GPU. Each GPU then answers about 6 requests a second within a 50 ms per-token target, so 50 requests a second needs about 11 GPUs with burst headroom.

What is the difference between time to first token and time per output token?

Time to first token is how long a user waits before the answer starts. It is dominated by prefill, the single pass over the whole prompt. Time per output token is the gap between later tokens, set by one decode pass over the current batch. Users notice the first as a delay and the second as reading speed.

How accurate are these numbers for a real deployment?

They show how a deployment scales, not exact figures for any vendor. The model assumes the ideal roofline (adjustable under Assumptions), an empty request queue, and that prefill does not stall decoding. Real servers reach a fraction of peak, so measure your own hardware and use the planner to see which change matters most.

Why does int4 quantization help less than grouped-query attention here?

At the default brief a decode step at the chosen batch reads about 13.5 GB of weights but about 51 GB of KV cache, so the cache dominates. int4 shrinks only the weights, from 13.5 GB to 3.4 GB. GQA with 8 KV heads cuts the cache four times, so each step reads far less and the GPU can hold four times as many requests.

Related explainers

KV Cache — Why LLM generation caches keys and values, and what it costs
Quantization — Shrinking model weights to 8 and 4 bits
Continuous Batching — Serving many requests at once without idle slots
Decoder-Only Transformers — The architecture behind GPT and modern LLMs
FlashAttention — Exact attention with far less GPU memory traffic
Speculative Decoding — Drafting tokens cheaply and verifying them in one pass

References

Last updated October 25, 2026