Chunk size: too small vs too big

A retrieval system splits each document into chunks, pieces of a fixed number of tokens, and searches over the chunks. Chunk size looks like a minor setting, but it decides whether the answer to a question is found whole, cut in two, or buried.

1. 1
   The document is 2,048 tokens long, and the gold strip marks a 120-token answer at tokens 300 to 420. Each row cuts the document into chunks of one size with no overlap, so chunks start every 128, 512 or 1,024 tokens. The highlighted chunk is the one holding most of the answer. A search embeds each chunk as one vector, a list of numbers that stands for its meaning, and returns the chunks whose vectors sit nearest the question's.
2. 2
   With 128-token chunks a boundary falls at token 384, inside the answer. The best chunk holds 84 of its 120 tokens, 70%, and the other 30% sits in the next chunk. A search that returns only the best chunk hands the model part of the answer. Overlap, repeating the last few tokens of each chunk at the start of the next, keeps answers near a boundary whole, at the cost of more chunks.
3. 3
   With 512-token chunks the answer is whole and fills 23% of its chunk. With 1,024-token chunks it is still whole but fills only 12%. The chunk's single vector now mostly encodes the other 900 tokens, so it matches the question less well. Larger chunks also make a longer prompt for every chunk retrieved.