Design for 50,000 docs · 20 queries/s · under 2.0 s and $0.002 a query
Documents50,000
Tokens per document1,500
Queries per second at peak20
End-to-end latency target2.0 s
Cost target per query$0.002
Corpus tokens
75M
Chunks to index
200K
Generation GPUs
13
Retrieval (embed, search, rerank) takes 6.8 ms of the 1.59 s. Writing the answer token by token takes 92%, so a RAG system's latency is mostly generation, the subject of Serve an LLM.
Assumptions
GPU efficiency vs the ideal roofline100%
Generator
7B, fp16
The Serve an LLM model: Llama 2 7B shape, 4,096-token window
GPU
80 GB · 2 TB/s · 300 TFLOP/s
Illustrative, the same accelerator as Serve an LLM; wrong by the efficiency factor if real kernels fall short
Encoder passes
2 × params FLOPs/token
Embedding model and cross-encoder; a single query is max(compute, fp16 weight read)
Search node
50 GB/s
Assumption: a CPU streaming vectors from RAM. Halve it and search time doubles
IVF overhead
8 B id · fp32 centroids
Per vector and per list; no compression such as product quantization
Toy set
2,000 × 16-d, 12 clusters
Seeded; 50 queries; k-means runs 8 iterations
Reranker model
noise 0.05 → 0.015
Simulated: first-stage and cross-encoder score noise on the toy set
Price
$2.00/GPU-hour
Same example rate as Serve an LLM; set it in the last step
Burst headroom
1.3×
Spare GPUs for traffic peaks, as in Serve an LLM
Not modelled
queueing, network, DB overhead
Latency assumes an empty queue; search-node cost is left out of cost per query
Drag Documents across its range and watch the offline boxes change while the online bar barely moves. Ingest cost grows with the corpus, but a query only touches the few chunks it retrieves, so query latency depends on the corpus only through search. The latency bar is built from the later steps: change a setting there and this bar changes too. Generation GPUs is queries per second times GPU-seconds per query times 1.3 burst headroom, rounded up.
Step 2 of 6: Chunking
Chunk size, overlap, and whether the answer stays whole
End to end
1.59 s
Per query
$0.00028
Index
616 MB
Recall@5, toy
0.92
Design for 50,000 docs · 20 queries/s · under 2.0 s and $0.002 a query
Chunk size c (tokens)512
Overlap o (tokens)64
Chunks per doc
4
Chunks N
200K
Storage
1.37×
Answer in best chunk
100%
The best chunk holds the whole answer, and the answer is 23% of that chunk. Overlap of 64 tokens repeats 64 tokens at each boundary, so the corpus is stored 1.37 times over, counting each chunk at its full size c.
Assumptions
GPU efficiency vs the ideal roofline100%
Generator
7B, fp16
The Serve an LLM model: Llama 2 7B shape, 4,096-token window
GPU
80 GB · 2 TB/s · 300 TFLOP/s
Illustrative, the same accelerator as Serve an LLM; wrong by the efficiency factor if real kernels fall short
Encoder passes
2 × params FLOPs/token
Embedding model and cross-encoder; a single query is max(compute, fp16 weight read)
Search node
50 GB/s
Assumption: a CPU streaming vectors from RAM. Halve it and search time doubles
IVF overhead
8 B id · fp32 centroids
Per vector and per list; no compression such as product quantization
Toy set
2,000 × 16-d, 12 clusters
Seeded; 50 queries; k-means runs 8 iterations
Reranker model
noise 0.05 → 0.015
Simulated: first-stage and cross-encoder score noise on the toy set
Price
$2.00/GPU-hour
Same example rate as Serve an LLM; set it in the last step
Burst headroom
1.3×
Spare GPUs for traffic peaks, as in Serve an LLM
Not modelled
queueing, network, DB overhead
Latency assumes an empty queue; search-node cost is left out of cost per query
Set Overlap to 0 and drag Chunk size from 128 to 1,024. Chunks start every c − o tokens, so a document needs ⌈(T − o) ÷ (c − o)⌉ of them. The toy answer spans tokens 920 to 1,040, and each chunk's share of it is the length of their intersection. Storage counts every chunk at its full size c, so it slightly overstates the last, shorter chunk. Real splitters also cut on sentences and headings, covered in the chunking explainer.
Step 3 of 6: Embedding and Index Size
Vectors in memory, ingest time, re-index cost
End to end
1.59 s
Per query
$0.00028
Index
616 MB
Recall@5, toy
0.92
Design for 50,000 docs · 20 queries/s · under 2.0 s and $0.002 a query
Embedding dimensions d768
Vector precision
Embedding model size E (parameters)110M
Documents changed per day2.0%
Index memory
616 MB
Full ingest
1.3 min
Ingest cost
$0.042
Re-index per day
$0.00083
Embedding 102M chunk tokens at 2 × 110M FLOPs each takes 1.3 min on the ideal GPU, about $0.042. The index is the lasting cost: 614 MB of vectors held in RAM, which int8 would cut to a quarter of fp32.
Assumptions
GPU efficiency vs the ideal roofline100%
Generator
7B, fp16
The Serve an LLM model: Llama 2 7B shape, 4,096-token window
GPU
80 GB · 2 TB/s · 300 TFLOP/s
Illustrative, the same accelerator as Serve an LLM; wrong by the efficiency factor if real kernels fall short
Encoder passes
2 × params FLOPs/token
Embedding model and cross-encoder; a single query is max(compute, fp16 weight read)
Search node
50 GB/s
Assumption: a CPU streaming vectors from RAM. Halve it and search time doubles
IVF overhead
8 B id · fp32 centroids
Per vector and per list; no compression such as product quantization
Toy set
2,000 × 16-d, 12 clusters
Seeded; 50 queries; k-means runs 8 iterations
Reranker model
noise 0.05 → 0.015
Simulated: first-stage and cross-encoder score noise on the toy set
Price
$2.00/GPU-hour
Same example rate as Serve an LLM; set it in the last step
Burst headroom
1.3×
Spare GPUs for traffic peaks, as in Serve an LLM
Not modelled
queueing, network, DB overhead
Latency assumes an empty queue; search-node cost is left out of cost per query
Switch Vector precision between fp32, fp16 and int8. Each vector costs d × bytes per value, and IVF adds an 8-byte id per vector plus one fp32 centroid per list. Ingest counts about 2 FLOPs per parameter per token, the same rule Serve an LLM uses for prefill, at the GPU's peak times the efficiency in Assumptions. Daily re-index cost assumes every changed document is re-chunked and re-embedded in full, which is how most pipelines handle an edit.
Step 4 of 6: Exact vs Approximate Search
An IVF index built and measured in the browser
End to end
1.59 s
Per query
$0.00028
Index
616 MB
Recall@5, toy
0.92
Design for 50,000 docs · 20 queries/s · under 2.0 s and $0.002 a query
n_list (k-means centroids)64
n_probe (lists scanned per query)4
k (results returned)5
Recall@5, toy
0.92
Scanned of 2,000, toy
141
Distances at 200K
12.6K
Search time
0.77 ms
At 4 of 64 lists, the toy index finds 92% of the true top 5 while scanning 141 of 2,000 vectors (the formula n_probe × N ÷ n_list gives 125; queries fall in denser lists). At 200K chunks that is 12.6K distances instead of 200K, 0.77 ms instead of 12 ms, 16 times less work.
Not modelled: HNSW, the graph index most vector databases default to, trades more memory for fewer distance computations. The vector search explainer builds one.
Assumptions
GPU efficiency vs the ideal roofline100%
Generator
7B, fp16
The Serve an LLM model: Llama 2 7B shape, 4,096-token window
GPU
80 GB · 2 TB/s · 300 TFLOP/s
Illustrative, the same accelerator as Serve an LLM; wrong by the efficiency factor if real kernels fall short
Encoder passes
2 × params FLOPs/token
Embedding model and cross-encoder; a single query is max(compute, fp16 weight read)
Search node
50 GB/s
Assumption: a CPU streaming vectors from RAM. Halve it and search time doubles
IVF overhead
8 B id · fp32 centroids
Per vector and per list; no compression such as product quantization
Toy set
2,000 × 16-d, 12 clusters
Seeded; 50 queries; k-means runs 8 iterations
Reranker model
noise 0.05 → 0.015
Simulated: first-stage and cross-encoder score noise on the toy set
Price
$2.00/GPU-hour
Same example rate as Serve an LLM; set it in the last step
Burst headroom
1.3×
Spare GPUs for traffic peaks, as in Serve an LLM
Not modelled
queueing, network, DB overhead
Latency assumes an empty queue; search-node cost is left out of cost per query
Set n_list to 256 and n_probe to 1, then raise n_probe. IVF clusters the vectors with k-means and stores each in the list of its nearest centroid. A query compares itself with every centroid, then scans only the n_probe nearest lists. Recall@k is the share of the true k nearest neighbours, found by comparing every vector, that the index returns. Search time assumes each comparison streams one vector from RAM at 50 GB/s.
Step 5 of 6: Top-k and Reranking
Context length, a cross-encoder, prefill and decode
End to end
1.59 s
Per query
$0.00028
Index
616 MB
Recall@5, toy
0.92
Design for 50,000 docs · 20 queries/s · under 2.0 s and $0.002 a query
Top-k chunks in the prompt5
Reranker
Cross-encoder size R (parameters)110M
Answer length (tokens)150
Answers decoded together per GPU4
Context tokens
2,790
Rerank
6.0 ms
Prefill
125 ms
Decode
1.46 s
The cross-encoder reads the question with each of 15 candidates, 6.0 ms of GPU time. In the simulation it lifts recall@5 from 0.69 to 0.85. Prefill of 2,790 tokens takes 125 ms; decoding 150 tokens at 9.7 ms each, 4 answers at a time, takes 1.46 s.
Assumptions
GPU efficiency vs the ideal roofline100%
Generator
7B, fp16
The Serve an LLM model: Llama 2 7B shape, 4,096-token window
GPU
80 GB · 2 TB/s · 300 TFLOP/s
Illustrative, the same accelerator as Serve an LLM; wrong by the efficiency factor if real kernels fall short
Encoder passes
2 × params FLOPs/token
Embedding model and cross-encoder; a single query is max(compute, fp16 weight read)
Search node
50 GB/s
Assumption: a CPU streaming vectors from RAM. Halve it and search time doubles
IVF overhead
8 B id · fp32 centroids
Per vector and per list; no compression such as product quantization
Toy set
2,000 × 16-d, 12 clusters
Seeded; 50 queries; k-means runs 8 iterations
Reranker model
noise 0.05 → 0.015
Simulated: first-stage and cross-encoder score noise on the toy set
Price
$2.00/GPU-hour
Same example rate as Serve an LLM; set it in the last step
Burst headroom
1.3×
Spare GPUs for traffic peaks, as in Serve an LLM
Not modelled
queueing, network, DB overhead
Latency assumes an empty queue; search-node cost is left out of cost per query
Raise Top-k until the prompt bar crosses the 4,096-token line. A cross-encoder reads the question and one chunk together and outputs a relevance score, so it costs a full forward pass per candidate, 3k of them here. The simulation adds random noise to each toy distance: more for the first-stage score, less for the cross-encoder's. Answers decoded together per GPU is the decode batch: each extra answer shares the weight reads but adds its own cache reads.
Step 6 of 6: Design Review
The design against the brief, and what can go wrong
End to end
1.59 s
Per query
$0.00028
Index
616 MB
Recall@5, toy
0.92
Design for 50,000 docs · 20 queries/s · under 2.0 s and $0.002 a query
✓End to end ≤ 2.0 s1.59 s
✓Cost per query ≤ $0.002$0.00028
✓Prompt fits the window ≤ 4,0962,940 tokens
·Index memory 200K × 768 fp32616 MB
·Re-index per day 2.0% of docs$0.00083
·Generation GPUs 20 queries/s × 1.313
At risk · Re-index every
Stale index: up to 1,000 changed documents (2.0%) are answered from old vectors before the next run.
Clear · Chunk size and overlap
Answer kept whole: one chunk holds all of the toy answer, which is 23% of that chunk.
Clear · Top-k
Context in bounds: 5 chunks make 2,790 tokens, inside the 4,096-token window and under 10 chunks.
Re-index every (hours)24
GPU price ($ per hour)$2.00
GPU-seconds per query
0.497
Per 1,000 queries
$0.276
Stale docs, worst case
1,000
Both targets are met: 1.59 s end to end and $0.00028 a query, almost all of it generation. The index costs 616 MB of RAM and $0.00083 a day to keep fresh, so freshness is limited by how often re-indexing runs, not by its price.
Assumptions
GPU efficiency vs the ideal roofline100%
Generator
7B, fp16
The Serve an LLM model: Llama 2 7B shape, 4,096-token window
GPU
80 GB · 2 TB/s · 300 TFLOP/s
Illustrative, the same accelerator as Serve an LLM; wrong by the efficiency factor if real kernels fall short
Encoder passes
2 × params FLOPs/token
Embedding model and cross-encoder; a single query is max(compute, fp16 weight read)
Search node
50 GB/s
Assumption: a CPU streaming vectors from RAM. Halve it and search time doubles
IVF overhead
8 B id · fp32 centroids
Per vector and per list; no compression such as product quantization
Toy set
2,000 × 16-d, 12 clusters
Seeded; 50 queries; k-means runs 8 iterations
Reranker model
noise 0.05 → 0.015
Simulated: first-stage and cross-encoder score noise on the toy set
Price
$2.00/GPU-hour
Same example rate as Serve an LLM; set it in the last step
Burst headroom
1.3×
Spare GPUs for traffic peaks, as in Serve an LLM
Not modelled
queueing, network, DB overhead
Latency assumes an empty queue; search-node cost is left out of cost per query
Lower Re-index every until the stale-index note clears. GPU-seconds per query is query embedding + rerank + prefill + answer tokens × decode step ÷ decode batch, since each decode step serves the whole batch at once. Cost per query is that times the GPU price per second. The search node runs on a CPU, and its cost is left out. Each note names the slider that controls it, so you can trade one failure for another.
How do you size a RAG system?
Retrieval-augmented generation (RAG) answers a question by first searching a document collection for relevant passages and then passing them to a language model as part of its prompt. A RAG system is two pipelines. An offline one splits documents into chunks, turns each chunk into an embedding vector and stores the vectors in an index. An online one embeds each question, searches the index, optionally reranks the results and generates the answer.
This case study sizes both pipelines for 50,000 documents of about 1,500 tokens and 20 queries a second, with a 2 second budget and a cost target of $0.002 a query. Generation uses the same 7B model, illustrative 80 GB GPU and prefill and decode model as the Serve an LLM case study, so the two agree. The search step runs a real inverted file (IVF) index on 2,000 vectors in the browser and measures its recall.
Key ideas
A RAG system is an offline ingest pipeline (chunk, embed, index) and an online query pipeline (embed, search, rerank, prefill, decode).
Chunk size trades precision for completeness: small chunks can cut an answer in two, large chunks keep it whole but bury it among other text.
Index memory is chunks × dimensions × bytes per value, so int8 vectors need a quarter of the memory of fp32.
An IVF index compares a query with n_list centroids and n_probe lists instead of every vector, at the cost of some recall that has to be measured.
Every retrieved chunk lengthens the prompt, which adds prefill time and can overflow the model's context window.
With a 7B generator, writing the answer takes most of the latency and cost, and retrieval takes a few milliseconds.
Common questions
What dominates RAG latency?
Generation. In the default design, embedding the question, searching 200,000 vectors and reranking 15 candidates take about 7 ms together. Prefill of the 2,790-token prompt takes 125 ms, and decoding a 150-token answer takes 1.46 s. Shorter answers, or decoding fewer answers together on each GPU, move the total far more than a faster index.
How much memory does a vector index need?
Multiply the number of chunks by the embedding dimensions and the bytes per value, then add the index's own overhead. 200,000 chunks of 768 fp32 values take 614 MB, and IVF adds an 8-byte id per vector and its centroids, about 1.8 MB here. Storing int8 values cuts the vectors to 154 MB, usually with a small loss of accuracy.
What is a good chunk size for RAG?
One large enough to hold a typical answer whole, with some overlap so answers near a boundary are not cut, and small enough that the answer is a large share of the chunk. A few hundred tokens is a common starting point. Measure recall on labelled questions from your own documents, because the best size depends on how they are written.
Is a reranker worth its latency?
Often, because it costs little next to generation. A 110M-parameter cross-encoder scoring 15 candidates of 512 tokens costs about 6 ms on the illustrative GPU. In this case study's simulation it lifts recall@5 from 0.69 to 0.85, but that gain is an assumption of the simulation; measure it on your own data before relying on it.