Agentic RAG vs Classic RAG: 2x to 4.4x the Cost per Correct Answer

Agentic RAG cost 2x to 4.4x as much per correct answer as single-shot retrieval in all 18 multi-hop benchmark cells we priced. The accuracy gain is often large where one-shot retrieval scores badly, and it fades near zero where one-shot already scores well. So the useful question about agentic RAG is how much your first pass is missing.

Agentic RAG vs classic RAG: the short answer

Agentic RAG usually answers more multi-hop questions correctly than single-shot retrieval when the first pass scores badly. It always costs more tokens. Published tests put the token multiple at 2.3x to 16.5x. Where one-shot F1 already reached 0.75, the gain in one direct comparison looked like noise.

I would not buy the loop first. I would test a different reader and a better single retrieval pass, then add iteration only for the questions that still fail. The rest of this post shows the numbers behind that order, and where they could mislead you.

What counts as agentic here

Classic RAG retrieves once, then generates. Agentic RAG lets the model decide whether to retrieve again, what to search for next, and when to stop. The label covers a wide range, from a fixed reason-then-retrieve loop such as IRCoT to a tool-using agent that can search, open a document and summarise it. Most academic baselines sit at the simple end, and Microsoft's system sits at the complex end.

Why multi-hop is the fair test

A multi-hop question needs two or more facts that live in different documents, so the second search depends on what the first one found. That is the case a single retrieval pass is structurally bad at. HotpotQA, 2WikiMultiHopQA and MuSiQue are the standard datasets. If the loop cannot justify itself there, it will struggle on simple lookups. Retrieval quality before the loop matters too, and the benchmarks on hybrid search and rerankers cover that cheaper lever.

Published agentic RAG benchmarks, in one table

This is every direct comparison we could open and read that reports a quality figure next to a cost figure. We also include the most cited vendor claim that does not. No two rows share a dataset, reader and token definition, so compare within a row and never across rows.

RASER agentic column is its IRCoT* reimplementation, and its ranges are the lowest and highest cells, not a single pairing. Multiples are our division of the published figures. Microsoft's FinanceBench accuracy row is labelled Traditional RAG and its token row Single-shot Search, and the paper never says they are the same system. Figures read 6 November 2026.

The vendor numbers omit the bill

Microsoft's Azure post is titled "Up to 40% better relevance for complex queries". Its table shows answer relevance moving from 41.55 to 57.63, about 39% in relative terms, as judged by GPT-4o. The post calls the cost and latency minimal, and it publishes no token count or latency figure. You cannot compute a cost per correct answer from it.

Microsoft's research paper is more candid. It reports a 5.9x improvement in recall@1 for 2.6x the tokens on BRIGHT. It also says its search stack trades raw retrieval quality for speed and scale, so the single-shot baseline starts weak by design. That is a fair production baseline, and a flattering one.

A revision flipped one result

EvoCtx is the clearest warning about how soft these numbers are. Version 2 of the paper, dated January 2026, printed IterDRAG below naive RAG on MultiHop-RAG, at 47.0% against 49.2%. Version 3, dated August 2026, prints it above, at 58.4% against 45.6%. The token counts stayed near 18,000 and 1,100.

Version 2 also printed identical HotpotQA cells for IterDRAG and RAG, which looks like a copy error. Version 3 shows different values. We use version 3 and name it, because a sign change between drafts of one paper tells you how much weight a single table can carry.

The agentic RAG gain shrinks as one-shot accuracy rises

The RASER preprint from June 2026 is the most useful table we found. It runs one-shot RAG and an iterative baseline side by side across 6 readers and 3 datasets, and it reports F1 and tokens in the same cells. We use all 18 cells, because picking 2 readers would tell a much tidier story than the table supports.

In the 10 cells where one-shot F1 was below 0.5, 8 gained at least 0.131. In the 5 cells between 0.6 and 0.7, the gain ran from 0.036 to 0.092. In the 3 cells at 0.75 or above, F1 moved by at most 0.027. With 200 to 500 questions per cell I would call those ties. Tokens still rose at least 2.3x in those cells.

My reading is that the loop mostly recovers what the first pass missed. So there is less to gain once the first pass is good. Dataset difficulty is tangled up in this, because the lowest scores cluster on MuSiQue and 2Wiki. This is one table from one preprint, so treat it as a hypothesis worth testing on your own data.

A larger model is not automatically the fix

One-shot RAG on GPT-OSS-120B outscored IRCoT on Llama-3.1-8B on all three datasets, on under a third of the tokens. The scores were 0.488 against 0.376, 0.754 against 0.532, and 0.777 against 0.676. Gemma-3-31B did the same at 0.393, 0.630 and 0.787. But Mistral-Small-119B, close to GPT-OSS in size, won only on HotpotQA, and it gained 0.216 and 0.208 from the loop elsewhere. So test specific readers on your data rather than assuming size settles it. Larger models also cost more per token or more to host, which this comparison does not capture.

Agentic RAG cost per correct answer, worked out

Accuracy and cost are normally reported apart, which is how a 9-point gain and a 3x bill get discussed as separate facts. Dividing one by the other gives you a single number. Cost per correct answer equals tokens per question, multiplied by your blended token rate, divided by the share of answers that are right.

RASER reports F1, not share correct, so we use F1 as the proxy and flag the problem below. For dollars we use Claude Sonnet 5.5 list rates of $2 input and $10 output per million tokens, blended at 90% input to $2.80 per million. That is a pricing unit for comparison, not what these open-weight readers cost to run.

The 90/10 blend is cautious. Hi-Q's cost table shows reader traffic at 94% to 99.8% input across the iterative methods it lists. So a real loop may cost less than this blend implies, particularly once prompt caching discounts the repeated context.

Working for Llama-3.1-8B on 2Wiki: 954 tokens divided by 0.307 is 3,107 tokens per correct answer, and 4,000 divided by 0.532 is 7,519, a ratio of 2.42x. Added: (4,000 minus 954) divided by (0.532 minus 0.307) is 13,538 tokens, or $0.038. Ratios do not change with price if the input and output mix is the same on both sides.

The added answer is the number to budget

Average cost per correct answer flatters the cheap option, because it ignores the questions the cheap option got wrong. The better figure is what each additional correct answer costs. In the table that ranges from about 3 cents to about 29 cents at Sonnet 5.5 rates. At gpt-4o-mini rates of $0.15 and $0.60, a $0.195 blend, the same range is about 0.2 cents to 2 cents.

Scale it. On Llama-3.1-8B and 2Wiki, 1,000 questions yield about 225 more correct answers for about $8.53 more at the Sonnet rate. If a wrong answer costs your support team one escalation, that is easy to justify. On GPT-OSS-120B, two of the three datasets show that you pay and receive nothing measurable.

Token counts for one method differ tenfold

Do not treat any multiple here as a property of agentic RAG. RASER records its IRCoT reimplementation at 2,500 to 5,400 tokens per question. Hi-Q reports IRCoT at 43,701 input tokens and 74 output tokens per question, across 2.92 LLM calls. Same named method, roughly ten times the tokens, and neither paper fully explains what is counted.

That gap decides the answer. Put Hi-Q's count into the Llama-3.1-8B 2Wiki cell and the added correct answer jumps from about 4 cents to about 53 cents. That mixes two papers, so it is a sensitivity check and not a finding. The practical lesson is that how much context you resend on each hop drives the bill more than the word agentic does. That matches the wider evidence on agent token variance.

Routing beat always-on agentic retrieval

The strongest result in this literature is older and less fashionable than agents. Adaptive-RAG, from KAIST and listed for NAACL 2024, sends each question to no retrieval, single-step or multi-step, based on predicted complexity. Its numbers make the case against running the loop on everything.

On HotpotQA, always-on multi-step retrieval scored 56.54 F1 at 9.38x the time of single-step. Adaptive-RAG with its real classifier scored 53.82 at 5.99x. With a perfect oracle router it scored 64.00 at 2.77x. Running the loop on questions that do not need it appears to hurt accuracy as well as cost. The same pattern shows on 2Wiki at 62.30 F1 and 1.69x time.

The gap between 53.82 and 64.00 is what a better classifier is worth. It is larger than the gap between single-step and always-on. This is also why I distrust vendor claims of large routing savings until I see the quality measured alongside them. RASER itself reports a routed version scoring 0.787 on HotpotQA at 1.8k tokens with GPT-OSS-120B, which beats both baselines in that cell. A paper's own method still deserves the most scepticism.

Where this rag cost per query argument is weakest

Three problems with this analysis could change the conclusion, and you should weigh them before you quote any number from it.

F1 is not the share of answers that were right

F1 gives partial credit for overlapping tokens, so dividing cost by F1 understates the cost of a truly correct answer. Exact-match scores would be lower and would make every dollar figure higher. The ratio between methods should move less, but RASER publishes F1 in this table, so we cannot prove that. FinanceBench reports answer correctness directly and gives 2.06x per correct answer, but it depends on pairing two rows the paper does not equate.

The baselines are simplified and the authors have a horse in it

RASER marks its IRCoT and Self-Ask columns with an asterisk, and its limitations section says some baselines were simplified versions. It is a preprint marked under review, and its authors are proposing a competing method. A faithful agent with better tools could beat these results. Microsoft's own system reached 92% on FinanceBench, close to its 94% oracle, and nothing in the academic tables tests an agent of that kind.

The benchmarks are not your corpus

Wikipedia-based multi-hop datasets reward a narrow skill. The peer comparison Is Agentic RAG worth it? tested non-multi-hop data and found that neither design wins everywhere. Agentic scored 55.6 to Enhanced RAG's 52.8 average NDCG@10 on query rewriting, then lost 43.9 to 49.5 on document list refinement, at up to 3.6x the cost. Your documents, your questions and the price of a wrong answer may put you on a different row. Nothing here measures that.

When agentic RAG earns its token bill

The evidence does not say never. It says the loop is a specific purchase for a specific condition, so here is how I would decide. The rows below are my judgement from the data above, not a published framework.

Where a wrong answer is expensive, the last row gets more attractive. A 30-cent added answer is trivial against a bad contract summary and absurd against a password reset. Put the price of a wrong answer next to the added-answer figure before you argue about architecture.

Frequently asked questions

Where to start this week

Pull 100 real multi-hop questions from your logs and label the correct answers. Run them through your current pipeline and record the input tokens, output tokens and exact correctness for each one. That gives you your own baseline, and the guide to building an eval suite covers how to keep it after this week.

Then run one test before any agent framework goes in: swap the reader for another model, keeping retrieval unchanged. Divide the extra cost by the extra correct answers. That number is the price the loop has to beat, and most teams have never written it down.

Related reading