RAG vs Long Context: We Priced a 10,000-Document Knowledge Base Both Ways

A 10,000-document knowledge base does not fit in a 1M-token context window. At an assumed 2,000 tokens a document it is 20 million tokens, so reading all of it per question costs $40.13 on Claude Sonnet 5.5 at list price. Retrieval augmented generation answers the same question for under 3 cents. On cost, RAG vs long context is not a close contest.

The 10,000-document knowledge base we priced

Most RAG vs long context comparisons price a single long prompt against a single short one. That framing quietly assumes your knowledge base fits in the window. For a real company knowledge base, it usually does not.

So this model starts from the corpus, not the prompt. Picture a support and policy library of 10,000 documents: help articles, contracts, runbooks, product specs. That is a modest base for a mid-sized software company.

The assumptions, stated

Every figure below rests on five assumptions. They are ours, not a vendor's, and you should replace them with your own before trusting any total.

Rates are list prices from Anthropic's pricing page as read on 9 October 2026. Month length is 30 days. No volume discounts, no Batch API, no data residency uplift.

Why a 1M-token window is not enough

Claude Sonnet 5.5, Opus 5.5 and Haiku 5.5 all carry a 1M-token context window, per Anthropic's context window documentation. Gemini 3.1 Pro and OpenAI's GPT-6.1 Sol sit in the same range.

A 1M window therefore holds about 500 of your 10,000 documents. To answer from the whole base without retrieval, you must split it into 20 shards, send the question to each, and merge the answers. That is the long-context architecture this post prices, because it is the only one that genuinely skips retrieval.

The alternative people actually build is one window holding a hand-picked 5% of the base. That is cheaper, and it is also a retrieval system. Someone, or something, chose which 500 documents went in.

Long context pricing on three official rate cards

The vendors price long prompts very differently. Anthropic dropped its long-context premium in March 2026. OpenAI and Google still charge more once a prompt crosses a threshold.

Sources: Anthropic, OpenAI and Google API pricing pages, read 9 October 2026. Above each threshold the whole request is billed at the higher rate. The final column uses each model's below-threshold rate, which is why it does not scale with the others.

Where the surcharges start

The thresholds matter more than the headline rates. On OpenAI's pricing page, GPT-6.1 Sol doubles its input rate from $2 to $4 above 272,000 tokens, and output rises from $10 to $15. Gemini 3.1 Pro doubles from $2 to $4 above 200,000 tokens.

Claude Haiku 5.5 is the odd case. It is 5 times dearer above 100,000 tokens, yet still the cheapest 1M prompt on the table at $0.50. Anthropic states that a 900k-token request is billed at the same per-token rate as a 9k-token request on its other current models.

Notice the final column. Below the threshold, all three flagship models charge the same $0.02 for a 10,000-token RAG prompt. Retrieval keeps you in the cheap tier on every vendor, which is a pricing advantage nobody advertises.

The RAG cost model at three query volumes

Here is the same knowledge base priced four ways at 1,000, 10,000 and 100,000 questions a month. Every rate traces to a vendor pricing page. Every multiplication is shown in the assumptions above.

Cached rows assume one full cache write per day at the one-hour write rate: $80 a day for the full base, $4 a day for one window. Low traffic with gaps over an hour will force more writes. RAG includes a Voyage rerank-3 call at its published estimate of $0.0025 a request and Pinecone's $50 Standard minimum.

What RAG actually costs to run

The retrieval side is cheaper than most teams assume. Embedding the entire 20M-token base with voyage-4-lite or OpenAI's text-embedding-3-small costs $0.40 at $0.02 per million tokens. Voyage's current models include 200 million free tokens, so the first full embed can cost nothing.

Storage is similarly small. At about 900 tokens a chunk, the base becomes roughly 22,000 vectors, a fraction of a gigabyte. Pinecone's Standard plan has a $50 monthly minimum and a $20 flat Builder tier, and pgvector on a Postgres you already run adds close to nothing. The full comparison is in the Pinecone, Weaviate and pgvector cost breakdown.

The generation call dominates. On Sonnet 5.5, 10,000 input tokens cost $0.02 and 500 output tokens cost $0.005. At 100,000 questions a month, model tokens are $2,500 of the $2,800 total. Our position is simple: spend your optimisation effort on the prompt, not the vector store.

Prompt caching changes the long context answer, but only for small corpora

Prompt caching is the strongest argument for long context. You pay once to write a large prefix, then pay a fraction to reread it. On Claude Sonnet 5.5, a cache read costs 0.05 times the base input price, so 1M cached tokens cost $0.10 instead of $2.

That 20-fold discount is real, and it is why the cached full-base row falls from $401,250 to $23,650 at 10,000 queries. It is still 73 times the RAG total, because reading 20 million cached tokens per question is still reading 20 million tokens.

The break-even rule for RAG vs long context

The useful question is not "which is cheaper". It is "at what corpus size does caching beat retrieval". The arithmetic is short. A cached corpus costs its size times the cache rate. A RAG prompt costs its size times the base rate.

They match when the corpus equals the RAG prompt divided by the cache multiplier. On Sonnet 5.5 that is 10,000 tokens divided by 0.05, or 200,000 tokens. On models with the standard 0.1 multiplier, such as Haiku 4.5 or Sonnet 4.6, it is 100,000 tokens.

Below 200,000 tokens, we would skip the retrieval stack entirely. You lose a moving part, a failure mode and an index to keep fresh. Above that line, every document you add raises the per-question cost of long context and leaves RAG untouched.

Two vendors make caching less generous than it looks. Gemini 3.1 Pro bills storage at $4.50 per million tokens per hour, so caching the full 20M-token base costs $64,800 a month in storage alone. OpenAI's cached reads on GPT-6.1 Sol double to $0.20 per million above 272,000 tokens. The wider pattern, where a falling unit price still produces a rising bill, is covered in the analysis of why cheap tokens keep raising AI bills.

The accuracy bill arrives after the invoice

Cost is only half the trade. The other half is whether the model actually uses what you paid to put in front of it. On that question, the published evidence is unkind to very long prompts.

The NoLiMa benchmark, published at ICML 2025, tested 13 models that claim at least 128K tokens of context. It hid facts whose wording barely overlapped the question, so models had to infer the link rather than match keywords. At 32K tokens, 11 of the 13 fell below half their own short-context score. GPT-4o dropped from 99.3% to 69.7%.

Newer models do better, and the vendors say so. Anthropic reported in March 2026 that Opus 4.6 scored 78.3% on MRCR v2 at 1M tokens, the highest among frontier models at that length. Read the other side of that number: the best reported model still missed more than a fifth of retrievals at full window.

That is the hidden cost of stuffing. You pay $40 for a question, and you buy a model attending to 20 million tokens with known blind spots. We covered the diagnosis side in the guide to fixing bad RAG before abandoning it, and the newer discipline of choosing what goes in the window in the piece on context engineering.

Where the case for RAG is weakest

This post argues for retrieval on cost. Here is where that argument loses, stated as plainly as the rest.

Long context wins on quality when you can afford it

The best controlled comparison we found favours long context on accuracy. Li and colleagues at Google DeepMind and the University of Michigan tested three models and concluded that, when resourced sufficiently, long context consistently outperforms RAG on average. The study is from 2024 and used models two generations old, which limits how far it transfers.

Their Table 1 shows the size of the gap. Gemini-1.5-Pro scored 49.70 with long context and 37.33 with RAG; GPT-4o scored 48.67 against 32.60. The same paper found RAG and long context gave identical answers on 63% of queries, which is the opening for a hybrid.

Be careful with these figures. The paper's own text and tables disagree in places, for example quoting a 65% cost cut for Gemini where Table 1 implies about 62%. We use the table values throughout.

The engineering cost this model leaves out

Our RAG column prices tokens and storage. It does not price the engineer who builds chunking, tunes retrieval, maintains an evaluation set and debugs a missed document at 2am. For a small team, that salary can dwarf a few thousand dollars of tokens.

Two smaller caveats cut the same way. Anthropic notes its newer tokenizer produces about 30% more tokens for the same text, so your base may be 26 million tokens on Claude, not 20. And our 2,000-token document is an assumption; a base of short FAQ entries could fit in one window.

A RAG architecture for 2026 that uses both

The framing of RAG or long context is the wrong question. The cost model says retrieval should decide what enters the window, and the large window should give retrieval room to be generous.

In practice, that means a three-step design. Retrieve 30 to 50 candidate chunks, rerank them, and send the top 20 or so into a prompt of 20,000 to 40,000 tokens. That prompt stays below every vendor's surcharge threshold and costs $0.04 to $0.08 in input on Sonnet 5.5.

Then add a router for the hard cases. When the model reports low confidence, escalate that one question to a cached long-context call over the relevant document set. Most questions take the cheap path, and the expensive path is reserved for the minority that need it, which is the pattern model routing applies to model choice.

Finally, measure it. A retrieval system you cannot score is one you cannot tune, and building an evaluation suite is the cheapest insurance in this whole design. Without it, you will not know whether a missed answer came from retrieval or from the model.

Frequently asked questions

Where to start this week

Measure your corpus before you choose an architecture. Run your actual document set through your target model's token counter and write the total down. If it is under 200,000 tokens, try a cached prompt first and keep the vector database off the roadmap.

If it is larger, take your current monthly question volume and multiply it by the per-query figures in the table above. Then compare that total with one engineer-month. Whichever number is bigger tells you where your next optimisation hour belongs.

Related reading