LLM Kitchen
Orders keep coming. Each step reads the whole model once, however many orders are cooking, so cooking several at once shares that cost. Find how many you need to keep up.
Set things up, then start the shift. You can change settings while it runs.
How many orders the GPU cooks at the same time. Each one is a burner in the kitchen below.
Line 0
GPU · up to 1 at once
Every step adds one token to every order on the GPU.
Served 0
One decode step: 21 ms
Each step makes one token for every order on the GPU: 47 tokens per second right now.
How to play
Orders are requests to a language model. Your GPU cooks a batch of them together, adding one token to each per step. Pick the settings, start the 60-second shift, and pass it by serving 90% of orders without making anyone wait too long. You can change settings mid-shift. Every player gets the same rush of orders.
What each shift teaches
- How many at once? Each step reads every weight once, so a batch shares that cost. Each order adds a little work, but far less than a step of its own.
- Dinner rush: static batching leaves finished slots empty, and no batch size can make up for it. Continuous batching refills them.
- Running out of memory: the KV cache trades memory for speed, and 8-bit weights free the memory it needs.
What's real and what's scaled
The shape of the costs follows real decoding: each step reads all the weights, the per-order math grows with the batch, the KV cache grows with every token in flight, and 8-bit weights halve both memory and the bytes read per step. The numbers are scaled for a 60-second game, so read them as proportions, not as measurements of a real GPU.
Common questions
Why does batching make LLM serving faster?
Each decode step reads all of the model's weights from memory once, however many requests are in the batch. A bigger batch shares that cost, so it produces more tokens per second, although each step takes a little longer.
What is continuous batching?
With static batching, the server waits for every request in a batch to finish before starting the next one, so short replies leave slots idle. Continuous batching refills a slot as soon as its request finishes.
What does the KV cache do?
It stores the keys and values the model already computed for earlier tokens, so each new token doesn't redo that work. The price is memory, which grows with every token of every request in flight.
Why quantize the weights?
Storing weights in 8 bits instead of 16 halves their memory, which leaves room for more KV cache. Decoding mostly waits on reading weights from memory, so fewer bytes per step also makes each step faster.