RAG latency in 1 picture

Retrieval-augmented generation (RAG) answers a question by searching a document collection first and handing the best passages to a language model. Most of the effort in building one goes into the search. Drawn to scale, the search is almost invisible.

1. 1
   The bar is one query through the default design of the Build a RAG System case study: 50,000 documents cut into 200,000 chunks of 512 tokens, a 7B model writing a 150-token answer, and an illustrative 80 GB GPU. Five stages run in order. Embed turns the question into a vector. Search finds the nearest chunk vectors in an inverted file (IVF) index. Rerank scores 15 candidates with a cross-encoder, a model that reads the question and a chunk together. Prefill reads the prompt, and decode writes the answer one token at a time.
2. 2
   The top bar is the whole query, 1.59 s. Prefill reads a 2,790-token prompt of five chunks, the question and a system prompt in one pass: 125 ms. Decode takes 1.46 s, about 92% of the total, because each of the 150 answer tokens needs its own pass that re-reads the model's 13.5 GB of weights. Four answers share each pass here, so each token takes 9.7 ms.
3. 3
   The lower bar enlarges the first 10 ms. Embedding the question takes 0.11 ms. Searching takes 0.77 ms, because IVF compares the query with 12,564 vectors instead of all 200,000. Reranking is the slowest retrieval stage at 5.96 ms, and all three together take 6.8 ms. A faster index would barely change what the user waits for.