Fine-Tuning vs RAG: What 5 Studies Found and What the Training Bill Costs

Retrieval beat plain fine-tuning in 3 of the 5 published studies that test both methods on the same knowledge tasks. In the other 2, the best score came from combining them. So the fine tuning vs RAG question is usually framed wrong. They do different jobs, and the training bill is the smallest part of what a fine-tune costs you.

Fine-tuning vs RAG: what the 5 studies actually tested

Most comparisons of fine tuning vs RAG are vendor explainers with no measurements in them. This post uses only studies that put both methods on the same questions, with the same base model, and published the scores. We found 5 that meet that bar, plus 1 study on fine-tuning alone that explains the pattern.

Two definitions first, because the studies use them precisely. Fine-tuning means continuing to train a model's weights on your data, so the knowledge lives inside the model. Retrieval augmented generation, or RAG, means searching your documents at question time and pasting the best passages into the prompt.

The studies differ in one way that matters more than any other. Some fine-tune on raw text, which is called unsupervised or continued pretraining. Others fine-tune on generated question and answer pairs, which is supervised fine-tuning. The second kind does better.

Our position, stated up front: the evidence does not support fine-tuning as a way to teach a model your facts. It does support fine-tuning as a way to teach a model how to behave around your facts.

The study-by-study fine-tuning vs RAG results table

The table below gives each study's own numbers, in the units the authors used. Scores are accuracy unless stated. Where a study tested many models, we show one representative row and say which.

Columns are not comparable across rows: each study used its own metric and question set. The RAFT "Both" figure is RAFT itself, which fine-tunes the model to answer from retrieved documents. Balaguer and Mecklenburg report gains in text and charts rather than in a single comparable table row.

Ovadia: retrieval wins on old and new facts

Ovadia and colleagues at Microsoft tested 3 open 7B models on 5 MMLU subjects and on a set of current-events questions the models could not have seen. On current events, regular fine-tuning made Llama2 7B worse, dropping it from 0.353 to 0.219. Retrieval lifted the same model to 0.585.

The authors found one partial fix. Fine-tuning on 10 paraphrases of each passage raised Mistral 7B to 0.588, and accuracy rose with the number of paraphrases. That still left it nearly 30 points behind retrieval.

Soudani: the less popular the fact, the bigger the gap

Soudani, Kanoulas and Hasibi tested 12 models from 80 million to 11 billion parameters on questions about little-known entities. Their abstract states that fine-tuning helps at every popularity level, but retrieval wins by a large margin on the least popular facts. That is exactly the kind of fact your internal knowledge base is full of.

Combining both did not reliably help in their setup. For Mistral 7B, adding tuning to retrieval lowered PopQA accuracy from 80.25 to 78.44. For the smaller StableLM2, it raised it from 76.14 to 82.09.

Mecklenburg: even GPT-4 with good training data trailed retrieval

This is the strongest test of fine-tuning in the set. The team used supervised tuning on GPT-4, generated 10 question and answer pairs per extracted fact, and still wrote that in no configuration did fine-tuning outperform RAG. Their finding on data design is useful anyway: generating questions fact by fact gave even coverage, while scaling by token count left about 20% of facts with no training example at all.

Balaguer: the two gains stack

The agriculture study by Balaguer and colleagues is the one vendors quote when they want fine-tuning to look good. Its abstract reports an accuracy gain of over 6 points from fine-tuning, which is "cumulative with RAG, which increases accuracy by 5 p.p. further." Read correctly, that is an argument for doing both, not for choosing tuning.

RAFT: fine-tuning that teaches the model to use retrieval

RAFT, from Berkeley, fine-tunes a model on questions paired with a mix of relevant and distracting documents. On PubMedQA it reached 73.30, above plain tuning at 59.7 and above GPT-3.5 with retrieval at 71.60. It is the clearest evidence that the best use of fine-tuning is to make retrieval work better, not to replace it.

Why fine-tuning struggles to store new facts

A separate Google study explains the pattern above. Gekhman and colleagues fine-tuned PaLM 2 on questions, some of which the model already knew and some it did not. They found that examples carrying new knowledge "are learned significantly slower" than familiar ones.

The worse finding is what happens once the model does learn them. As the unfamiliar examples are fitted, they "linearly increase the model's tendency to hallucinate". Trained on facts it did not know, the model's test accuracy fell from 37.5 at early stopping to 25.8 when trained to convergence.

The practical reading is blunt. Pushing your knowledge base into the weights harder does not make the model know it better. It teaches the model to answer confidently about things it never really learned.

This is also why we think "fine-tune it on our docs" is the most expensive misunderstanding in enterprise AI projects. The training run is cheap. The wrong answers that follow, delivered in a fluent house style, are not.

The training bill on OpenAI and Google rate cards

Both providers bill training per token processed. Google's tuning documentation states that training tokens equal the tokens in your dataset multiplied by the number of epochs. OpenAI applies the same arithmetic. So the bill is a function of three numbers: dataset size, epochs and rate.

How a knowledge base becomes training tokens

A 1-million-token knowledge base is roughly 750,000 words, a mid-sized library of help articles and policies. Raw text trains poorly, as Ovadia showed, so a serious tune turns it into question and answer pairs. Mecklenburg's team scaled its datasets to 10 times the source tokens and trained for 3 epochs.

Applied to our 1-million-token base, that gives a 10-million-token dataset and 30 million training tokens per run. That is the assumption every figure below rests on. Change it and the bill scales in a straight line.

Inference prices are input / output per 1M tokens. OpenAI figures are from its pricing page, read 3 November 2026. Google lists tuning at $0.01 and $0.003 per 1,000 for Gemini 3.5 Flash and Flash-Lite, converted here to per million. Google's documentation says inference pricing is the same for each stable Gemini version, which we read as no tuned-model premium. Generating the training pairs adds about $16 on GPT-4.1 mini or $80 on GPT-4.1, for 10 million output tokens.

For comparison, embedding the same 1-million-token base for retrieval costs 2 cents with text-embedding-3-small, or 13 cents with the large model. Re-embedding after an update costs the same again. On upfront cost alone, retrieval is between 1,000 and 37,000 times cheaper than a tune.

Neither figure includes the work that actually costs money: building an evaluation set, checking the generated training pairs, and testing the result. Those are measured in engineer weeks, and they apply to both paths.

A fine-tuning vs RAG cost model for one knowledge base

Upfront cost is the wrong comparison, because the two methods spend money at different times. Fine-tuning pays once per training run and then sends short prompts. RAG pays almost nothing upfront and then sends long prompts on every question. So the real question is volume.

The model below assumes 100,000 questions a month, 500 tokens of instructions and question per prompt, and 300 output tokens per answer. Retrieval adds 5 chunks of 500 tokens, so a RAG prompt is 3,000 tokens. The knowledge base changes monthly, so a tuned model is retrained monthly.

Zan Digital cost model, list rates read 3 November 2026. Excludes vector database hosting, which depends on vendor and index size, and all engineering time. Gemini embedding cost assumed comparable to OpenAI's small model. No prompt caching applied, because retrieved chunks differ per question.

Two results stand out. First, on OpenAI the fine-tuning premium eats most of the saving from shorter prompts. A tuned GPT-4.1 mini answer costs $0.00136 against $0.00168 with retrieval, a gap of about a third of a tenth of a cent.

Second, Google's pricing changes the shape of the trade. With no tuned-inference premium, a tuned Gemini 3.5 Flash answer costs less than half a RAG answer. If cost were the only variable, Gemini would be the place to fine-tune.

Why the cost win does not settle it

Every cheap row in the table is a fine-tuned model answering without retrieval. That is the configuration that lost in 3 of the 5 studies above. Saving $400 a month on Gemini is not a saving if the tuned model gets your refund policy wrong a third more often.

The configurations the evidence supports, base with RAG or tuned with RAG, are the more expensive rows. That is the honest answer to "what does fine-tuning cost": if you do it properly, it is an addition to your retrieval bill, not a replacement for it.

The OpenAI fine-tuning cost now includes an expiry date

Here is the part of the training bill that no rate card shows. A fine-tuned model is tied to its base model, and the base model has a shutdown date set by the vendor. Your tune is an asset with a useful life you do not control.

OpenAI made this concrete in 2026. Its pricing page now states that "OpenAI is winding down the fine-tuning platform" and that it "is no longer accessible to new users." The schedule on its deprecations page runs as follows.

Inference on existing tunes continues until the base model is retired. For GPT-4.1 and GPT-4.1 mini, that is 14 January 2027, with GPT-6 Luna named as the replacement. There is no way to carry a tune across. You rebuild it on whatever the vendor offers next, if it offers anything.

Google is not immune either. Its model page says Gemini 2.5 Flash was retired from its enterprise platform on 20 October 2026. The tuning-capable list there has already moved on to the 3.x Flash models priced above.

Retrieval has no equivalent problem. An index of embedded chunks works with any model that accepts a prompt. Switching the model behind a RAG system is a configuration change and a regression test, which is a point we make at more length in the analysis of what model switching really costs. Switching a fine-tuned system is a new project.

So amortise any tune over the life of its base model, not over an imagined three years. A $750 GPT-4.1 run in November 2026 has about 10 weeks of serving left. That is the real OpenAI fine tuning cost in late 2026.

Where the case for RAG is weakest

The argument above has real gaps.

The studies mostly used small, older models

3 of the 5 studies tested models of 7 billion parameters or fewer, from 2023 and early 2024. Only Mecklenburg and Balaguer tuned GPT-4, and neither priced the work. None tested a 2026 frontier model, and none of the 4 models in our cost table appears in any study. We are applying old accuracy results to new prices, and that is a stretch.

Several studies also used an ideal or very good retriever. Soudani's headline table assumes ideal retrieval. Your retriever will be worse, and every point it loses comes off the RAG column. Chunking alone moves results a lot, as the benchmarks in our review of RAG chunking strategies show.

Some jobs genuinely belong to fine-tuning

None of these studies measured what fine-tuning is best at. They measured factual recall. A tuned model that always returns valid JSON, writes in your support team's voice, or follows a 12-step triage policy without a 3,000-token prompt is solving a different problem. The RAFT results on API calls show tuning adding over 60 points where the job was learning a syntax.

Latency is the other honest point in tuning's favour. A 500-token prompt is answered faster than a 3,000-token one. For a voice agent or an autocomplete feature, that can decide the architecture regardless of the accuracy studies.

And the counting is ours. The "3 of 5" tally reflects which studies we judged comparable. A different reader could add studies we excluded and get a different ratio.

When to fine-tune an LLM, and when to retrieve

Retrieve for facts, tune for behaviour, and do both only when you have proved you need both. The table sets it out, including the row where tuning clearly wins.

Before either, measure whether you have a retrieval problem at all. Many teams that ask about fine-tuning have a retriever returning the wrong chunks. Fix that with hybrid search and reranking, compared in our benchmarks of rerankers and hybrid search, before you pay to change any weights.

If your corpus is small, you may need neither. A knowledge base under a few hundred thousand tokens can often sit in a cached prompt, which we priced in the RAG vs long context cost model.

Frequently asked questions

Where to start this week

Write 50 real questions your users ask, with the correct answers, before you spend anything on either method. Run them against a base model with your current retrieval and record the score. If you have no retrieval yet, that is the first build. Our guide to building an LLM evaluation suite covers how to score the answers.

If you already run a fine-tuned OpenAI model, look up its base model's shutdown date today. If it is GPT-4.1 or GPT-4.1 mini, you have until 14 January 2027, and the replacement plan is probably retrieval on a newer base model rather than another tune.

Related reading