From Sanskriti Khandelwal | Product & Market Analysis
RAG Is Not Dead. Your Retrieval Is Just Bad, and the Benchmarks Say Where
On this page
Long context windows did not retire retrieval augmented generation. They moved the failure from the model to the pipeline feeding it. On the NoLiMa benchmark, 11 of 13 long-context models fell below half their short-context accuracy at 32,000 tokens. RAG is not dead. Most RAG is simply unmeasured, and the fixable part sits in chunking and ranking.
Key takeaways
- Every model tested got worse as its input grew. Chroma evaluated 18 models across four vendor families and found degradation in all of them, including on a task as simple as replicating repeated words.
- Retrieval beat context stuffing on the same model. Llama3.1-70B scored 34.26 F1 on EN.QA reading roughly 117,000 tokens of full context, and 44.43 F1 reading 16,000 tokens of ordered retrieval.
- Default chunking retrieves the right passage and a great deal of noise. The common recursive splitter measured 88.5% recall at 7.0% precision, so most of the text reaching the model was irrelevant.
- Ranking changes more than embedding choice does. Contextual embeddings, keyword search and a reranker together cut top-20 retrieval failures from 5.7% to 1.9%, a 67% reduction.
What long context actually changed
Two things changed, and only one of them is about capability. Anthropic's platform documentation now lists a 1M-token context window as the default on its current models, billed at standard rates across the full window.
The documentation is explicit about the arithmetic. A 900,000-token request is billed at the same per-token rate as a 9,000-token request. The surcharge that used to punish long prompts is gone.
That is a genuine change. It removes the pricing reason to keep prompts short, and it makes "put the whole document in the prompt" a plausible engineering choice for the first time.
A window is a capacity, not a guarantee. The same Anthropic documentation says so directly: as token count grows, accuracy and recall degrade, a behaviour it names context rot.
Read that in context. The vendor selling the 1M-token window is telling you that curating what goes into it matters as much as how much space is available.
Retrieval is that curation step. Declaring it dead because the window grew is like declaring database indexes obsolete because disks got cheaper. The capacity went up. The cost of scanning everything did not go away.
The evidence that long windows degrade
Three published results carry this argument. None of them comes from a vector database vendor with a product to defend, and one comes from a model provider with the opposite incentive.
NoLiMa removed the literal match
Most needle-in-a-haystack tests are easier than they look. The needle and the question usually share words, so a model can find the answer by matching strings rather than by understanding the passage.
NoLiMa, published by Adobe Research in February 2025 and presented at ICML 2025, removed that shortcut. Its needles and questions have minimal lexical overlap, so the model must infer a latent association to locate the answer.
The paper evaluated 13 models that claim support for at least 128K tokens. All performed well below 1,000 tokens. At 32,000 tokens, 11 of the 13 dropped below half their short-context baseline. GPT-4o, one of the stronger performers, fell from 99.3% to 69.7%.
Note what that is not. It is not a claim that long context is useless. It is a claim that the advertised window and the usable window are different numbers, and the gap widens as the answer gets less literal.
Context rot, measured across 18 models
Chroma published a technical report in July 2025 covering 18 models across the Anthropic, OpenAI, Google and Alibaba families. The finding is uncomfortable and consistent. Every model tested got worse as input length grew.
That held even on trivial tasks. In a repeated-word replication exercise, performance fell steadily as the sequence grew from 25 words to 10,000. Nothing about that task requires reasoning.
Two secondary results matter more than the headline. Retrieval degraded further when the target sat in the middle of a long input. And shuffling the haystack into random sentence order improved accuracy across all 18 models, compared with the original coherent document.
That second result should unsettle anyone who assumes a model reads a document the way you do. Structure that helps a human reader appears to hurt retrieval from a long window.
The same work isolated distractors, meaning passages that are semantically close to the query but do not contain the answer. Adding a single distractor reduced accuracy against a clean baseline. Adding more compounded the effect, and it grew worse at longer inputs.
This is the finding that maps straight onto a production pipeline. Your retriever's job is not only to find the right chunk. It is to keep the near-misses out, and most systems are never measured on that second job at all.
Retrieval still wins on the same model
The cleanest comparison holds the model fixed and changes only what enters the prompt. In Defense of RAG in the Era of Long-Context Language Models did exactly that in September 2024.
Using Llama3.1-70B on the EN.QA long-context benchmark, the authors compared full-context input against order-preserve retrieval. That variant keeps retrieved chunks in their original document order rather than sorting them by relevance score.
| Setup | Input tokens | F1 score |
|---|---|---|
| Llama3.1-70B, full context, no retrieval | About 117,000 | 34.26 |
| Llama3.1-70B, order-preserve RAG | 16,000 | 44.43 |
| Llama3.1-70B, order-preserve RAG | 48,000 | 47.25 |
Figures from the paper's EN.QA results. This is one model on one benchmark from September 2024, so treat the ranking as directional rather than as a settled result across every model since.
The retrieval configuration scored higher while reading roughly one seventh of the tokens. The authors also documented an inverted U-shape: answer quality rises as more chunks are retrieved, peaks, then falls as irrelevant chunks crowd the window.
That curve is the practical point of this whole post. There is an optimal amount of context for your system, it is smaller than your window, and the only way to find it is to measure. Most teams have never plotted their own version of that curve.
Diagnose your retrieval before you rewrite it
Most teams respond to bad answers by swapping the model or rewriting the prompt. Both changes are visible and quick. Neither is usually the cause.
Measure recall and precision separately
Recall asks whether the right passage was retrieved at all. Precision asks what share of the retrieved tokens were relevant. They fail differently and they need different fixes.
If recall is low, the model never saw the answer, and no prompt change will help. If recall is high and precision is low, the model saw the answer buried in near-misses, which is precisely the condition the distractor research says degrades output quality.
Teams that report a single "accuracy" number have collapsed two different problems into one, and then argue about the model.
Test the retriever without the model
Run the retrieval step alone against a labelled set and score it. You need 50 to 100 real queries with the correct source passage marked by a human who knows the domain.
That set is the asset. It is cheap to build, it survives model upgrades, and it converts "the answers feel worse this month" into a number you can act on. It belongs alongside the rest of your testing, which is covered in the walkthrough on building an evaluation suite and in the tests that separate a pilot from production.
| What you observe | Most likely cause | What to measure | First fix to try |
|---|---|---|---|
| Confidently wrong answers about specifics | A near-miss passage was retrieved instead of the right one | Precision at k on labelled queries | Add a reranking stage before generation |
| Exact identifiers, error codes and part numbers are missed | Embedding-only retrieval | Recall on identifier queries specifically | Add keyword search alongside vector search |
| Answers degrade as the corpus grows | Rising distractor density | Precision at k tracked over time | Retrieve wide, rerank narrow |
| The answer is cut off mid-fact | Chunk boundaries splitting a claim | Recall at k, plus manual review of 20 chunks | Increase chunk size or add overlap |
| A passage is clearly retrieved and clearly ignored | Too much context, wrong position | Tokens sent per query | Cut the context and preserve document order |
Chunking is where most quality complaints start
Chunking is the least glamorous decision in the stack and the one with the widest measured spread. It is usually made once, by a default parameter nobody chose.
The defaults are not neutral
Chroma's chunking evaluation, published July 2024, tested strategies across five corpora including WikiText, ConvFinQA and PubMed Open Access. The spread between the best and worst strategy reached 9 percentage points of recall on the same corpus.
The precision figures are the ones nobody quotes. With text-embedding-3-large and five chunks retrieved, the standard recursive character splitter at 200 tokens scored 88.5% recall and 7.0% precision.
Read the second number again. On that configuration, roughly 93% of the text handed to the model had nothing to do with the question. You pay for those tokens once at the meter and again in accuracy, because the distractor research says they actively hurt.
Semantic chunking is not automatically better
The intuitive fix is to split on meaning rather than on character count. The evidence for that is weaker than its popularity suggests.
In Chroma's set, the model-driven chunker reached the highest recall at 91.7% and the worst precision at 3.9%. A separate study, Is Semantic Chunking Worth the Computational Cost?, evaluated semantic against fixed-size chunking across three retrieval tasks and concluded that the computational costs are not justified by consistent performance gains.
My position, stated plainly: start with a well-parametrised recursive splitter and spend the saved effort on ranking. Semantic chunking is a second-order optimisation being sold as a first-order one, and it is the most common place I see teams burn a quarter.
The exception worth respecting is document structure. If your corpus is paginated filings, tables or code, splitting on the document's own boundaries beats splitting on a token count. That is not semantic chunking. That is reading the format, and it starts with getting the underlying data into a retrievable state.
Ranking is the other half of the problem
Chunking decides what can be found. Ranking decides what actually arrives in the prompt. The published gains are larger on the ranking side, and they are measured against a single clear metric.
Anthropic reported results for a three-part change in September 2024, using top-20 chunk retrieval failure rate as the target. Each stage is additive and each was measured separately.
Contextual embeddings alone cut the failure rate from 5.7% to 3.7%, a 35% reduction. The technique prepends a short model-written description of where each chunk sits inside its parent document, then embeds the combined text. The stated one-time cost was $1.02 per million document tokens.
Adding contextual BM25 took it to 2.9%, a 49% reduction. BM25 is keyword matching, and it belongs in that stack for a specific reason. Embeddings are weak on exact strings. Error codes, SKUs, ticket references and function names are where vector search quietly fails, and where a keyword index costs almost nothing to add.
Adding a reranker took it to 1.9%, a 67% reduction against the original. The configuration retrieved the top 150 chunks and reranked down to 20 before generation.
The reranker is the highest-yield single change in that sequence, and it is not free. It adds a service call and latency to every request. Anthropic says so in the same post, which is more candid than most vendor benchmarking, and it is the trade you should price before you ship. Once it is live, treat ranking quality as a monitored signal rather than a launch decision, in the same way you would handle drift in a production agent.
One more constraint gets missed at this stage. Ranking has to respect who is allowed to see what, and retrieval systems that ignore permissions surface documents the asker was never entitled to read. That failure mode is examined in the comparison of enterprise search permission models.
The cost argument, done honestly
The standard claim is that retrieval is cheaper than long context. That was more true when long prompts carried a surcharge. It is a weaker argument in 2026 than it was in 2024, and the honest version needs the caveats attached.
Do the arithmetic yourself. Retrieving 8,000 tokens instead of stuffing 200,000 is 25 times fewer input tokens per query. At Claude Sonnet 5's published rate of $2 per million input tokens, that is $0.016 against $0.40 per query. Across 100,000 queries a month the gap is $1,600 against $40,000.
Then apply the counter-argument. Prompt caching changes the picture wherever the same large document is queried repeatedly. A cache hit costs 10% of the standard input price on Anthropic's published multipliers, which turns that $0.40 into $0.04 for the cached portion.
So the shape of your workload decides the answer. One document and many questions favours stuffing plus caching. A corpus that changes daily, with questions that span documents, defeats the cache and returns you to retrieval.
The point I would hold onto: the token bill is the second argument for retrieval, not the first. Accuracy is the first. Teams that justify RAG on cost alone tend to abandon it the moment a vendor cuts prices, which is the wrong reason to change architecture. The wider pattern of falling unit prices and rising bills is covered in the piece on cheap tokens and rising AI bills. What it does to a software P&L sits in the analysis of inference inside cost of goods sold.
| Condition | Retrieve | Fill the window |
|---|---|---|
| Corpus larger than the window | Required | Not possible |
| Corpus changes daily | Strong fit, index updates incrementally | Cache invalidates constantly |
| Answers need permission filtering per user | Filter at retrieval time | No natural place to enforce it |
| Evidence spread evenly through one document | Fragments the reasoning | Clear winner |
| One document, many repeated questions | Adds machinery for little gain | Clear winner with prompt caching |
The last two rows are the concession. There are real workloads where building a retrieval pipeline is the wrong call, and pretending otherwise is how vector databases end up in systems that never needed one.
Where this argument is weakest
The case above is built on other people's benchmarks. Here is what that costs it.
Long context genuinely wins for some tasks
When evidence is spread evenly across a document and the question needs most of it, retrieval fragments the reasoning. Summarising a contract, comparing two filings section by section, or tracing a change across a codebase are cases where the whole document in the window is the right call.
The OP-RAG result cuts both ways here too. Its own inverted U-shape means that beyond a point, more retrieved context helps. The argument is about where the peak sits, not about whether retrieval always wins.
The benchmarks are not your corpus
Every number in this post comes from a published benchmark on a public corpus. Your documents have different structure, different jargon and different query patterns.
Chroma's evaluation used five corpora and the strategy rankings were not identical across them. Treat these figures as evidence that the choices matter, not as a configuration to copy into your pipeline on Monday.
The dates matter, and we have no first-party data
The Anthropic retrieval figures are from September 2024 and the chunking evaluation from July 2024. Embedding models and rerankers have moved since then. The direction of those findings has held up in later work, and the exact percentages should be read as dated.
Zan Digital also publishes no proprietary retrieval benchmark. Every figure above is someone else's measurement on someone else's corpus. That is the weakest thing about this post. It is also why the diagnostic section tells you to build your own labelled set rather than trust ours.
Frequently asked questions
Is RAG dead in 2026?
No. Long context windows changed the trade-offs rather than removing the need for retrieval. Benchmarks published in 2025 show accuracy falling as input grows: on NoLiMa, 11 of 13 long-context models dropped below half their short-context baseline at 32,000 tokens. Retrieval is the step that decides what enters the window. A bigger window makes that choice cheaper to get wrong, not unnecessary.
Should I use RAG or a long context window?
Use retrieval when the corpus is larger than the window, changes often, or needs permission filtering per user. Use the full window when one document must be reasoned over as a whole, such as a contract review or a section-by-section comparison. The published evidence supports a hybrid: retrieve to narrow the candidate set, then give the model room to reason over what survives. Measure both on your own queries first.
Why does my RAG system give confidently wrong answers?
Usually because the retriever returned a near-miss rather than nothing. Chroma's 2025 context rot research found that a single semantically similar distractor reduced accuracy against a clean baseline, and that the effect compounds with more distractors and longer inputs. The model is not inventing an answer from thin air. It is answering from the wrong passage that your ranking stage put in front of it.
What chunk size should I use for RAG?
There is no single answer, and the evidence says the choice matters more than most teams assume. Chroma's chunking evaluation found up to 9 percentage points of recall separating the best and worst strategies on the same corpus. Start with a recursive character splitter at a few hundred tokens, measure recall and precision on your own labelled queries, then adjust. Copying a number from a blog post is not measurement.
Does reranking actually improve RAG accuracy?
Yes, and it is the highest-yield single change in the published sequence. Anthropic measured top-20 retrieval failures falling from 5.7% to 2.9% with contextual embeddings and keyword search, then to 1.9% once a reranker was added. That is a 67% reduction overall. The cost is latency and an extra service call on every query, which Anthropic states in the same post.
How do I know whether retrieval or the model is the problem?
Score the retrieval step on its own. Take 50 to 100 real queries, have a human mark the passage that contains the correct answer, then measure how often your retriever returns it. If the right passage is missing, no prompt or model change will fix the answer. If it is present and the answer is still wrong, the problem is downstream.
Where to start this week
Three measurements, in order, none of which requires changing a line of pipeline code.
First, build the labelled set. Pull 50 real queries from your logs, have someone who knows the domain mark the passage that answers each one, and store it in version control. This takes an afternoon and it is the only thing on this list you cannot skip.
Second, score recall against that set. If the correct passage is absent from the retrieved chunks more than one time in ten, stop tuning prompts. Your problem is upstream of the model and every hour spent on prompt wording is wasted.
Third, count the tokens you send per query and estimate how many of them are relevant. If that fraction looks anything like the 7% precision figure above, add a reranking stage before you touch anything else. It is one service call, it is measurable in a week, and it is the change with the largest published effect.
The related decision
Retrieval quality is one input to a bigger question about whether a system is ready to ship. The tests that answer it are in the production readiness checklist, and the harness that runs them is in the evaluation suite walkthrough.
References
- Modarressi et al., NoLiMa: Long-Context Evaluation Beyond Literal Matching, arXiv, February 2025, presented at ICML 2025. Used for the 13-model evaluation, the 32K figure and the GPT-4o baseline.
- Hong, Troynikov and Huber, Context Rot: How Increasing Input Tokens Impacts LLM Performance, Chroma, 14 July 2025. Used for the 18-model result, the distractor findings and the shuffled-haystack result.
- Smith and Troynikov, Evaluating Chunking Strategies for Retrieval, Chroma, 3 July 2024. Used for all recall and precision figures by chunking strategy.
- Anthropic, Contextual Retrieval in AI Systems, 19 September 2024. Used for the failure-rate sequence, the reranker configuration and the $1.02 per million token cost.
- Yu, Xu and Akkiraju, In Defense of RAG in the Era of Long-Context Language Models, arXiv, September 2024. Used for the EN.QA table and the inverted U-shape finding.
- Qu, Tu and Bao, Is Semantic Chunking Worth the Computational Cost?, arXiv, October 2024. Used for the semantic against fixed-size chunking conclusion.
- Anthropic, Context windows and Pricing, platform documentation accessed 28 August 2026. Used for the 1M-token default, the context rot statement and all token prices.
- Grand View Research, Retrieval Augmented Generation Market Size Report, 2025. Used only as background on category spend, not for any accuracy claim.
The weakest thing about this source base: four of the eight sources are dated 2024, and two of the three retrieval benchmarks were published by a vector database company. The NoLiMa result is the independent one, and it argues against long context rather than for any product.
Related reading