RAG Chunking Strategies: What 5 Benchmarks Found, and Where They Disagree

Chunk size alone moved Recall@1 from 4.5% to 71.5% on one dataset in a 2025 Fraunhofer IAIS study. Swapping the embedding model at the same chunk size moved it by under 10 points. Retrieval augmented generation chunking is not a preprocessing detail. Five published benchmarks agree on that, and disagree on almost everything else, including which method wins.

Short answer

Start with recursive or fixed-size chunks of 256 to 512 tokens and little or no overlap. Move smaller, toward 64 to 128 tokens, if your answers are single facts. Move larger, or chunk by page, if answers span paragraphs or live in PDFs. Treat semantic chunking as an experiment, not a default.

This post is for the engineer who owns a RAG pipeline and has to pick a splitter this sprint. It compiles 5 published evaluations, states each dataset and metric, and flags where they contradict each other. The wider case for fixing retrieval before swapping models is made in the breakdown of where RAG retrieval actually fails. This one stays narrowly on chunking.

The 5 RAG chunking benchmarks, side by side

Each benchmark below measures something different, so the numbers cannot be pooled into one ranking. Chroma measures how many retrieved tokens are relevant. Vectara and Fraunhofer measure whether the right document or answer was retrieved. NVIDIA measures whether the final answer was right, as judged by a language model.

That spread is the first finding. Two teams can both say "semantic chunking works" and mean different things, because one measured token precision and the other measured answer accuracy.

Four of the 5 studies were written by companies that sell retrieval or embedding products. Only the Fraunhofer IAIS paper is from a research institute with nothing in the pipeline to sell. Read the vendor rows as informed but interested.

Conflict one: does chunking or the model matter more?

Fraunhofer IAIS says chunking, on most datasets. Jina AI's own table says the model, by a wide margin. Both are right about their own setup. Fraunhofer compared two strong embedding models of similar quality, so the model swap moved little. Jina compared a small model against a much larger one, so the model swap moved a lot.

My reading is that chunking matters as much as the model once you are choosing between competent models. It does not rescue a weak one.

Conflict two: is semantic chunking worth it?

Chroma found its clustering chunker gave the best precision and an LLM chunker gave the best recall. Vectara found semantic chunking rarely beat fixed-size on real documents. The difference is the metric and the corpus. Chroma rewards cutting irrelevant tokens. Vectara rewards finding the right document, and on real documents fixed-size already did that well.

RAG chunk size: small for facts, large for context

The cleanest evidence on rag chunk size comes from Bhat and colleagues at Fraunhofer IAIS, published in May 2025. They split 6 question-answering datasets into fixed token chunks of 64, 128, 256, 512 and 1,024 tokens, with no overlap. They then retrieved with two embedding models, Stella and Snowflake Arctic, and checked whether the answer text appeared in the top chunks.

The result is not a single optimum. It is two opposite slopes, depending on what kind of answer the dataset contains.

Fact lookup wants small chunks

On SQuAD, where answers are short spans inside Wikipedia paragraphs, Stella scored 64.2% Recall@1 at 64 tokens and 38.6% at 1,024. A larger chunk dilutes the one sentence that matters with text the query never mentioned. The embedding then represents the paragraph's average topic, not the fact.

The authors' own summary is that "smaller chunks (64-128 tokens) are optimal for datasets with concise, fact-based answers." If users ask for a part number, a date or a policy limit, go small.

Technical support documents behave the other way. On TechQA, Stella scored 4.9% at 64 tokens and 61.9% at 1,024. NarrativeQA, built on long stories, also peaked at 1,024. When the answer is an explanation spread over several paragraphs, a small chunk holds only a fragment of it, and the retriever cannot see that the fragment belongs to the right answer.

Chunk size versus embedding model

The paper reports both models at every size, which makes it possible to compute something it does not state directly. For each dataset, I took the largest Recall@1 swing caused by changing chunk size, and the largest swing caused by switching model at a fixed size. The working uses the paper's published table, nothing else.

Chunk size produced the larger swing on 5 of 6 datasets. On TechQA it was 67.1 points against 9.6 for the model. That is the strongest published support for the claim that chunking choices move retrieval as much as model choice does. It is one paper, with two models, and the limits of that are covered further down.

One more caution about this paper. Its prose sometimes disagrees with its own tables. The text says Snowflake does better with smaller chunks, yet Snowflake's best Recall@1 sits at 1,024 tokens on 5 of the 6 datasets. Use the tables, not the summary sentences.

Fixed and recursive splitters hold up better than their reputation

Chroma's July 2024 technical report, by Brandon Smith and Anton Troynikov, is the most cited chunking evaluation. It treats chunking as a token-efficiency problem. It built 472 queries over 5 corpora, including a State of the Union address, Wikipedia text, chat logs, financial filings and PubMed papers. It then measured what share of retrieved tokens were actually relevant to each query.

That framing matters because a retriever can find the answer and still send the model a page of noise. Recall alone hides that cost.

Precision is the hidden cost of big chunks

The best recall came from an LLM chunker at 91.9%, and the best precision from Chroma's own clustering chunker at 8.0%. But the plain recursive splitter at 200 tokens with no overlap scored 88.1% recall and 7.0% precision, close to both leaders. The authors wrote that it "is consistently high performing across all evaluation metrics." I agree, and I would make it the default before trying anything that costs an LLM call per document.

The weakest configuration was the one many teams inherit without choosing it. An 800-token chunk with 400 tokens of overlap, which Chroma maps to the OpenAI Assistants default, scored 1.4% precision. Chroma called it "particularly poor" on the recall and efficiency tradeoff. At that setting, roughly 98 of every 100 retrieved tokens were irrelevant to the query.

Overlap also cost more than it gave in this test. Recursive chunks of 400 tokens with 200 overlap scored 88.1% recall, against 89.5% with no overlap. The authors note that overlap repeats tokens, which pushes precision and IoU down by construction.

Semantic chunking benchmark: the win only shows up on stitched documents

Semantic chunking splits text where the meaning shifts, usually by embedding each sentence and cutting where neighbouring sentences stop resembling each other. It costs an embedding call per sentence at indexing time. In an October 2024 preprint titled "Is Semantic Chunking Worth the Computational Cost?", Vectara researchers Renyi Qu, Forrest Bao and Ruixuan Tu tested whether that cost pays.

They compared fixed-size chunks, defined by sentence count, against breakpoint and clustering semantic chunkers. Every run used the same Stella embedding model. They scored document retrieval, evidence retrieval and final answers generated by gpt-4o-mini.

Stitched documents flatter semantic chunking

To create documents with clear topic shifts, the team stitched short documents from datasets such as Miracl and Natural Questions into long ones. On those, breakpoint chunking won by a wide margin: 81.9 against 69.5 F1@5 on Miracl, and 63.9 against 43.8 on NQ. That is the result vendors quote.

On the 4 original datasets, which were not stitched, fixed-size chunking scored highest every time. On HotpotQA it reached 90.6 against 87.4 for breakpoint. The authors note that stitched documents have "artificially high topic diversity," which is exactly the condition semantic splitting is designed to detect.

The answer-generation results were flatter still. Across 5 datasets, BERTScore differed by 0.01 at most between the three methods. The paper concludes that "fixed-size chunking remains a more efficient and reliable choice for practical RAG applications." That matches my reading of the table.

Real enterprise corpora sit between the two cases. A single policy PDF rarely jumps topic every few paragraphs, but a merged export of support tickets might. If your documents look like the stitched sets, semantic chunking deserves a test. If they look like ordinary reports, the evidence says you are paying for nothing.

Late chunking: a small, consistent gain with one condition

Late chunking reverses the usual order. Instead of splitting text and embedding each piece alone, it runs a long-context embedding model over the whole document first. It then splits the resulting token vectors at the chunk boundaries and averages each group. Each chunk embedding therefore carries context from the rest of the document, which matters when a chunk says "the company" without naming it.

Jina AI introduced the method in a September 2024 paper by Michael Gunther and colleagues, revised in July 2025. It needs no extra training, but it needs an embedding model with a long context window that uses mean pooling.

The gains were small and steady. Averaged over 3 embedding models and 4 BEIR datasets, late chunking lifted nDCG@10 by 1.8 points with fixed 256-token boundaries, 1.9 with sentence boundaries and 1.5 with semantic ones. It scored below naive chunking in 1 of 36 cells. The largest single gain was on NFCorpus with the small Jina v2 model, from 23.5 to 30.0.

Where late chunking does not help

The paper is candid about its limits. On synthetic needle-in-a-haystack tasks, the authors write that "late chunking is not useful, as the additional context from the document is totally irrelevant." On some reading-comprehension tasks, naive chunking did better with large chunks.

The bigger caveat sits in the paper's own main table. Every score from the small model with late chunking is lower than the larger Jina v3 model without it. The small model's average across the 4 datasets moved from 46.1 to 48.7 with late chunking, while the larger model scored 56.7 without it. Late chunking is a cheap improvement on top of a good model, not a substitute for one.

Page-level chunking for PDFs: NVIDIA's end-to-end result

NVIDIA's June 2025 evaluation, written by Steve Han, is the only one of the 5 that scores final answers rather than retrieval. It ran 7 strategies through one pipeline: token chunks of 128 to 2,048 tokens at 15% overlap, one chunk per page, and one chunk per layout section. Answers came from a Nemotron 70B model and were graded by two LLM judges.

Page-level chunking produced the highest average accuracy, at 0.648, with the lowest variance across datasets. Every token-based size landed between 0.603 and 0.645. The recommendation is to "start with page-level chunking as your default strategy."

The per-dataset picture is less tidy. On FinanceBench, 1,024-token chunks scored 0.579 against 0.566 for pages. On the Earnings set, 512 tokens led at 0.681. On KG-RAG, pages won at 0.520 while 128-token chunks scored 0.421. The average favours pages because pages never lost badly, not because they always won.

There is a practical point the scores do not capture. Pages are fixed boundaries, so a citation can say "page 14" and a reviewer can check it. For regulated teams, that alone can decide the question.

Where this argument is weakest

The headline claim of this post, that chunking moves accuracy as much as model choice, rests mainly on one study. Here is what cuts against it.

The model can still matter more

Fraunhofer IAIS compared two strong, similar embedding models. Jina AI compared a small model with a much larger one and found the model gap was about four times the late chunking gain. If your current embedding model is weak, upgrading it probably beats any chunking change. The comparison of embedding model cost and benchmark scores is the place to check that first.

Second, the metrics are not comparable. Token precision, F1@5, nDCG@10, string-matched Recall@1 and LLM-judged accuracy each reward different behaviour. Fraunhofer's string matching counts a chunk as correct if the answer text appears anywhere in it, which flatters large chunks. Nobody in this set measured the same pipeline on all five metrics.

Third, the sources are thin and partly interested. Chroma, Jina AI, NVIDIA and Vectara all sell retrieval, embedding or RAG products. Chroma's queries were generated by an LLM. None of the 5 studies were independently replicated that I could find. None reported indexing cost in numbers, though Chroma notes LLM chunking can take "tens of minutes."

Finally, every dataset is English and most are public benchmarks. Your contracts, tickets and wikis may behave differently. That is an argument for measuring your own corpus, which is the only test that settles anything.

RAG best practices the chunking evidence actually supports

Taken together, the 5 studies support a short set of defaults. Each row states how much evidence sits behind it, because some of these rest on a single test.

Overlap is the one setting where the studies point in different directions. Chroma found overlap hurt. Jina AI found a 16-token overlap helped naive chunking on 3 of 4 datasets. NVIDIA found 15% best among three values, in a single FinanceBench test it described as "not a full-grid search." Start at zero and add overlap only if your own recall improves.

Chunking is also only half of retrieval. Once the splits are sensible, ranking decides what reaches the model, and the evidence on hybrid search and rerankers shows gains of a similar size. Tune chunking first, because a reranker cannot recover a passage that was cut in half.

Frequently asked questions

Where to start this week

Pull 50 real questions from your logs and mark the passage that answers each one. That small set is enough to compare a 256-token recursive split against your current setting, and it will tell you more than any benchmark in this post. The steps for turning it into a repeatable test are in the guide to building an LLM eval suite. The tools that run it are compared in Promptfoo, DeepEval and Ragas side by side.

Then check one setting you probably never chose. If your pipeline still uses 800-token chunks with 400 tokens of overlap, change it first. That was the worst configuration Chroma measured.

Related reading