1Vector Search & HNSW 2Chunking & Reranking for RAG 3Model Calibration 4Data Drift & Model Monitoring

Lensa ML

Chunking & Reranking for RAG

Step 1 of 6: Why Chunk?

Retrieval returns pieces, not documents

Chunk3 of 6

chunk 3 · tokens 256–384
…museum in the world and holds the Mona Lisa. The museum opens at 9 am and stays open late on Fridays. It is closed every Tuesday, so plan your visit for another day. Booking a timed ticket online is the easiest way to skip the long queue at the glass pyramid. The Musee d'Orsay, in a former railway station, holds the largest collection of Impressionist paintings. It closes on Mondays instead of Tuesdays. Entry is free on the first Sunday of each month, but booking ahead is still required. [Food and drink] Bakeries sell fresh baguettes early in…

Guide length

691 tok

Chunks

6

This chunk

128 tok

Embedder limit

512 tok

Chunk 3 starts and ends in the middle of a sentence. Fixed-size splitting ignores the text, and 4 of the 5 boundaries here cut a sentence. The guide is longer than the 512-token embedder limit, and one vector for all 6 sections would blur them together. Each chunk gets its own vector, and retrieval returns chunks, not documents.

Drag Chunk from 1 to 6 and read each 128-token piece of the guide. Token counts here are estimated at about 1.3 tokens per English word. Most embedding models built on BERT-sized encoders accept 512 tokens and silently truncate anything longer, so the end of a long document would never be indexed. A real knowledge base also holds far more text than the 7B model's 4,096-token context. The original RAG paper indexed Wikipedia as 100-word passages, following the DPR retriever, and that granularity is still a common starting point.

Step 2 of 6: Chunk Size

Precise but cut, or whole but diluted

Chunk size96 tokens

Chunks

8

Top chunk score

0.52

Best other guide

0.47

Answer share

21%

A boundary at token 288 cuts the amber answer in two, so no chunk holds both "opens at 9 am" and "closed every Tuesday". Smaller chunks are precise, but they lose the sentence that says what "It" refers to.

Drag Chunk size from 32 to 512 tokens and watch the hit curve rise and then fall. The index here also holds nine simulated city guides, about 6,300 tokens of generic travel text, so the answer chunk always has competitors. Questions differ in what they need: a lookup such as an opening day wants a small chunk, while a summary question needs a large one. One common fix is small-to-big retrieval, which searches small chunks for precision and then passes their larger parent section to the model.

Step 3 of 6: Overlap

Repeat the edges so answers stay whole

Chunk size

Overlap0% (0 tokens)

Chunks

6

Storage

1.00×

Kept whole

10/12

Top-1 hits

10/12

With no overlap, 2 of the 12 answers straddle a boundary and no chunk holds them whole. Overlap also moves every boundary, so the top-1 hit count can dip between settings.

Pick a Chunk size of 64, 128, or 256 and drag Overlap from 0 to 50 percent. With overlap O, chunks start every L − O tokens, so storage grows by about L / (L − O): at 128 tokens and 25 percent overlap the stride is 96 and the factor is 1.33. Each extra chunk also costs an embedding call, and neighbouring chunks that share text can fill several top-k slots with near-duplicates. Many pipelines settle on 10 to 20 percent overlap and rely on better boundaries for the rest.

Step 4 of 6: Structure-Aware Splitting

Cut on sentences, paragraphs, and headings

Split on

Max chunk size96 tokens

Chunks

10 (avg 69)

Mid-sentence cuts

0

Kept whole

12/12

Top-1 hits

12/12

Whole paragraphs are packed up to 96 tokens, and a paragraph that is too long falls back to its sentences. Answers that live inside one paragraph stay together. Every answer fits whole in one chunk.

Click Fixed, Sentence, Paragraph, and Heading, then drag Max chunk size from 48 to 256. Recursive splitters, such as the one in LangChain, try the largest separator first, a blank line between paragraphs, and fall back to line breaks, then spaces, only when a piece is still too long. Markdown and HTML splitters use headings the same way. A further step is to prepend the section title to each chunk, so a passage that only says "It closes on Mondays" still mentions museums when it is embedded.

Step 5 of 6: Reranking

A fast first pass, then a careful second look

Question

Top-k from the bi-encoderk = 5

Bi-encoder rank

4

After rerank

1

Cross-encoder passes

5

The vector search puts the answer chunk at rank 4. The cross-encoder reads the question and each of the 5 chunks together and moves it to rank 1. Each candidate costs one cross-encoder pass, so k sets the latency. Scores are illustrative.

Pick a Question and drag Top-k from the bi-encoder from 1 to 10. A bi-encoder embeds the query and each chunk separately, so chunk vectors are computed once at indexing time and search is a nearest-neighbour lookup. A cross-encoder feeds the query and one chunk through the same transformer together, which lets attention compare their words directly. Nogueira and Cho (2019) used BERT this way to rerank BM25 candidates on MS MARCO. Production pipelines often retrieve 50 to 100 candidates and keep the reranker's top 3 to 5.

Step 6 of 6: Evaluating Retrieval

Recall@k and MRR on labelled questions

Pipeline

k for recall@kk = 3

Recall@3

75%

MRR@10

0.71

Answer at rank 1

7/12

Vector search alone gives MRR 0.71. It misses questions that hinge on an exact term such as "112" or "Navigo". Recall@3 counts the 9 questions whose answer chunk is in the top 3.

Click each Pipeline and drag k for recall@k from 1 to 10. MRR is (1/|Q|) Σ 1/rank_i. For vector search here the answer ranks are 1 for seven questions, then 2, 3, 4, 4, and 5, so MRR = (7 + 0.5 + 0.33 + 0.25 + 0.25 + 0.2) / 12 = 0.71. Labelling answer spans instead of chunk ids keeps the question set valid when the chunking changes. Reciprocal rank fusion comes from Cormack et al. (2009). For graded relevance, nDCG is the usual metric.

How do chunking and reranking work in RAG?

Chunking is the step in retrieval-augmented generation (RAG) that splits documents into short passages, each embedded and stored as its own vector. Reranking is a second retrieval pass that rescores the first set of results with a slower, more accurate model. Together they decide which text reaches the language model's prompt, so they set an upper bound on how good a RAG answer can be.

Every RAG system chunks its documents, whether it answers questions over support articles, legal contracts, code, or product manuals. The chunk size, the overlap, and where the boundaries fall all change what a search can find. Rerankers sit between vector search and the prompt in most production pipelines, because a cross-encoder catches relevant passages that a fast bi-encoder ranked too low. Teams tune both with a labelled question set and metrics such as recall@k and MRR.

Key ideas

RAG retrieves chunks, not whole documents, so the chunking decides what the model can be shown.
Small chunks match precisely but can split an answer, while large chunks keep it whole but dilute its embedding.
Overlap guarantees that any answer up to the overlap length fits whole in some chunk, at the cost of extra storage.
Splitting on sentences, paragraphs, and headings keeps related text together better than fixed token counts.
A cross-encoder reranker reads the query and each candidate together and fixes the order of a fast first-stage search, but it cannot recover a chunk that was never retrieved.
Recall@k and MRR on a labelled question set turn chunking and retrieval choices into measurable numbers.

Short takes on this topic

Chunk size: too small vs too big

Common questions

What chunk size should I use for RAG?

There is no single best size. Many teams start between 256 and 512 tokens with 10 to 20 percent overlap, then compare a few sizes on a labelled question set. Short factual questions tend to favour smaller chunks, and questions that need a whole section favour larger ones.

What is semantic chunking?

Semantic chunking embeds each sentence and starts a new chunk where the similarity between neighbouring sentences drops, on the idea that a drop marks a change of topic. It needs no fixed size and follows the content. It costs an embedding call per sentence at indexing time, and on well-structured documents it often performs about as well as heading-aware splitting.

Is a reranker worth the extra latency?

Often, yes. A cross-encoder over 20 to 100 candidates usually adds tens to a few hundred milliseconds on a GPU, while generating the answer takes longer. It helps most when the right chunk is retrieved but ranked below the cut-off for the prompt. If the right chunk is not in the candidate list at all, better chunking or hybrid search will help more.

What is the difference between BM25 and vector search?

BM25 scores documents by exact term matches, weighting rare terms higher and normalizing for document length. Vector search compares learned embeddings, so it matches paraphrases but can blur rare names, codes, and numbers. Hybrid search runs both and merges the two ranked lists, often with reciprocal rank fusion.

Related explainers

Retrieval-Augmented Generation — Ground language model outputs in retrieved documents to reduce hallucination
Embeddings — Turning words and categories into vectors
TF-IDF — Score how important a word is to a document in a collection — the backbone of search engines and text retrieval
BERT & Masked Language Modeling — Mask a word, predict it from both sides — how BERT learns deep bidirectional representations
Model Evaluation & Metrics — Precision, recall, F1, ROC curves, and regression metrics
Norms & Distance Metrics — L1, L2, cosine similarity — measuring closeness in vector spaces

References

Last updated October 25, 2026