From Aryan Vatsa | Product & Market Analysis
Model Routing in Production: 31.7% Cheaper, Not the 85% Everyone Quotes
On this page
The largest independent test of model routing measured a 31.7% cost reduction while matching the best single model in the pool. The most quoted figure in the category is 85%. Both are real results, they measure different experiments, and model routing pays out very differently depending on which task you point it at. This post measures the gap across five.
Key takeaways
- The 85% saving and the 31.7% saving are not the same experiment. RouteLLM cut cost 85% on MT Bench against GPT-4 alone while holding 95% of its quality. LLMRouterBench routed across 33 models and 21 datasets and reached 31.7% while matching the best single model.
- The quality gap is task-shaped, not size-shaped. A domain-trained small model beat five frontier models on contract extraction at 78% to 97% lower cost. The best small coder models reach roughly 50% on SWE-bench Verified against 80.8% for a frontier model.
- Below 4 billion parameters the bottleneck is instruction following, not intelligence. On the VAREX benchmark, models that echo the schema instead of filling it lose 45 to 65 percentage points of accuracy. Fine-tuning at 2B recovered 81 points.
- Compounding is what actually breaks routed agent workflows. At 95% per-step accuracy, a 10-step workflow finishes correctly 59% of the time. The per-call difference looks trivial and the end-to-end difference does not.
What model routing is, and what it is not
The short answer
Routing simple tasks to cheaper models typically saves 30% to 50% of inference spend, not 85%. Quality loss is close to zero on classification and short-form extraction, moderate on long-context summarisation, and severe on repository-level code and multi-step agent work. Measure the gap on your own traffic before you set any threshold.
Model routing sends each request to the cheapest model that can handle it, instead of sending everything to the most capable one. A classifier, a heuristic or a small model inspects the request and picks a tier.
Three mechanisms get called routing and they behave differently. A router decides once, before inference. A cascade runs the cheap model first and escalates when a confidence check fails. A fallback only fires on error or timeout, which means it never catches a wrong answer.
The distinction matters because it decides what you pay for a mistake. A router that guesses wrong pays once and ships a bad answer. A cascade that guesses wrong pays twice and ships a good one. Most published savings figures come from routers, and most production systems end up as cascades.
My position is that routing is a real cost lever with a modest ceiling, and teams budgeting against 85% are budgeting against a 2024 marketing number. The unit that should govern the decision is cost per successful task, not cost per call. Those two numbers move in opposite directions the moment retries enter the picture, and almost every published saving is quoted on the first one.
The two savings numbers everyone quotes disagree
In July 2024, LMSYS published RouteLLM. Its routers, trained on 55,000 Chatbot Arena preference samples, cut cost by 85% on MT Bench, 45% on MMLU and 35% on GSM8K while holding 95% of GPT-4 quality. Notice the spread inside that one paper. The same technique saved 85% on one benchmark and 35% on another.
The best RouteLLM router reached 95% of GPT-4 performance using 26% GPT-4 calls, and 14% after data augmentation. That result is genuine and it is also narrow. It compares a two-model setup against a single expensive baseline on conversational prompts.
In January 2026, a team from Shanghai AI Lab and Northwestern Polytechnical University published LLMRouterBench. It covers 391,645 instances across 23,945 prompts, 21 datasets and 33 models, at a stated cost of 1,000 GPU hours and $2,771.84 of API spend. The best routing methods delivered up to 31.7% cost reduction while matching the strongest single model.
The same benchmark scored a commercial router at 24.7% below its baseline. A router can lose you money and quality at the same time, and one shipping product did.
Neither team is wrong. RouteLLM measured how much you save by not calling one very expensive model on easy conversational prompts. LLMRouterBench measured how much you save when your alternative was already a sensible single model choice across a realistic task mix. The second question is the one a production team is actually asking.
The gap across five enterprise tasks
Aggregate benchmark scores hide the thing you need. What matters is whether the gap on your task is 2 points or 30. Here are five common enterprise workloads with the closest published measurement for each.
1. Intent and topic classification
This is the safest thing to route, and it is also the highest-volume thing in most agent stacks. NVIDIA researchers argued in June 2025 that agent work consists mostly of intent classification, data extraction and structured generation, tasks with clear boundaries that small models handle. Their position paper makes the case that small models are sufficient for most invocations in agentic systems.
The caveat is document length. Short-text topic classification depends on shallow lexical features that small models capture well. Classification over long legal or clinical text needs capacity the small models do not have, and that is where measured gaps open up.
2. Structured extraction from documents
This is the most surprising result in the set, and it cuts both ways. IBM researchers published VAREX in March 2026, a benchmark of 1,777 documents with 1,771 unique schemas across 20 models. Their finding is that below 4 billion parameters the bottleneck is structured output compliance, not extraction ability.
The specific failure has a name. Schema echo is when a model returns something shaped like your schema instead of the values you asked for. In affected models it cost 45 to 65 percentage points. Extraction-specific fine-tuning at 2 billion parameters recovered 81 points, which tells you the deficit is instruction following and is fixable without scale.
At the other end, a domain-trained small model can beat frontier models outright. A May 2026 paper reported a self-hosted legal mixture-of-experts model beating five frontier models. It reached a macro F1 of 0.812 and a micro F1 of 0.842, at 78% to 97% lower inference cost. It also recorded the highest precision in the study. Read that one with care, because the authors evaluated their own commercial model.
3. Summarisation over long context
Summarisation is where teams assume routing is safe and where it quietly is not. Retrieval and reasoning are different capabilities, and long-context benchmarks separate them. On the RULER benchmark, every model scores 10 to 25 points below its single-needle retrieval score at the same context length.
Model size makes that worse. Llama-3 models at 1 billion and 3 billion parameters nearly fail entirely at 128K context. A cheap model that summarises a 2,000 word ticket thread perfectly can fall apart on a 200 page contract, and the failure is a plausible summary that omits the clause you needed.
4. Code generation
Here the gap depends entirely on the unit of work. On function-level generation the small models have caught up. Directional figures compiled by Towards Data Science put Phi-4 at 14 billion parameters on 82.6 for HumanEval, ahead of Llama-3.3 at 70 billion on 78.9. Those figures carry no stated sample or collection date, so treat them as indicative.
On repository-level work the gap is wide. The same compilation puts the best 30 billion parameter coder models around 50% on SWE-bench Verified, against 80.8% for a frontier model. That is roughly 31 points, on the exact task an engineering team cares about most. The distance between benchmark coding and repository coding is the subject of the piece on where coding agents actually fail in real repositories.
5. Multi-step agent execution
This is the task where routing decisions compound, and the arithmetic is unforgiving. For a workflow of m independent steps with per-step error rate p, end-to-end success is (1 minus p) to the power m. At 99% per-step accuracy you finish 90.4% of 10-step runs and 36.6% of 100-step runs.
Drop per-step accuracy to 95% and a 10-step workflow succeeds 59% of the time. Drop it to 90% and you are at 35%. A cheap model that is 4 points worse per call is not 4 points worse per workflow. I would not route any workflow longer than five steps to a cheaper tier until per-step accuracy has been measured rather than assumed, and the case against multi-agent architectures rests on the same arithmetic.
| Task | What the evidence shows | Routing verdict |
|---|---|---|
| Intent and topic classification | Small models handle short-text classification adequately; long domain text opens a real gap | Route by default. Cap input length. |
| Structured extraction, flat schema | Fine-tuned 2B models recovered 81 points of accuracy on VAREX | Route, ideally to a fine-tuned model. |
| Structured extraction, nested schema | Schema echo costs 45 to 65 points below 4B parameters | Do not route until you validate output. |
| Long-context summarisation | RULER scores sit 10 to 25 points below single-needle retrieval at the same length | Route short inputs only. |
| Repository-level code | Roughly 50% against 80.8% on SWE-bench Verified, directional | Do not route. |
| Multi-step agent execution | 95% per-step accuracy yields 59% success at 10 steps | Route individual steps, never the plan. |
Rows draw on different benchmarks with different metrics, so the numbers are not comparable to each other. The comparable thing is the direction and rough size of each gap.
Where the savings actually come from
Routing works because list prices span two orders of magnitude. On a verified pricing table dated 31 July 2026, the cheapest listed general model runs at $0.10 per million input tokens and the most expensive at $10.00. Output tokens span $0.40 to $50.00.
| Model | Input | Output | Cached input |
|---|---|---|---|
| Claude Fable 5 | $10.00 | $50.00 | $1.00 |
| GPT-5.6 Sol | $5.00 | $30.00 | $0.50 |
| Claude Sonnet 5 | $2.00 | $10.00 | $0.20 |
| Claude Haiku 4.5 | $1.00 | $5.00 | $0.10 |
| GPT-5.4 mini | $0.75 | $4.50 | $0.075 |
| Gemini 2.5 Flash | $0.30 | $2.50 | $0.03 |
| GPT-5.4 nano | $0.20 | $1.25 | $0.02 |
| Gemini 2.5 Flash-Lite | $0.10 | $0.40 | $0.01 |
Work an example on those prices. Take 10 million input and 2 million output tokens a day. All of it on GPT-5.6 Sol costs $110 a day. Route 70% to GPT-5.4 nano and the bill falls to $36.15, a 67% saving.
Now assume 15% of the nano answers fail review and get retried on Sol. The retries add $11.55, the total becomes $47.70, and the saving drops to 57%. A retry rate you never measured is the difference between a 67% story and a 57% one, and every point of retry above that keeps eroding it.
The pricing table also shows the lever most teams skip. Cached input runs at roughly a tenth of standard input across every provider listed. If I had one afternoon and one cost decision to make, I would spend it on prompt caching rather than a router. Caching does not change which model answers, so it cannot degrade quality. Why falling token prices have not produced falling bills is covered in the analysis of cheap tokens and rising AI invoices.
One more asymmetry is worth naming. Output tokens cost 4 to 6 times input tokens at every tier, so a cheap model that rambles can cost more than an expensive model that is terse. Measure tokens produced per task, not just price per token, a point that connects directly to how inference cost lands on gross margin.
The failure mode a benchmark cannot show you
Compounding is not a rounding error
Every eval suite scores single calls. Every agent runs chains. A model that scores 95% on your golden set and a model that scores 99% look interchangeable on a dashboard and produce a 31 point difference over 10 steps.
LLMRouterBench found the related failure inside routing itself. On the 11.9% of queries where only one or a few models answered correctly, routers reached just 23% to 25% accuracy. Routers are weakest precisely where model choice matters most, which is the opposite of what you would want.
Silent regression is the operational risk
A router changes which model serves a request, and it can do that without raising an error. Quality falls, the error rate stays at zero, and nobody finds out until customers complain.
Anthropic published a postmortem in September 2025 describing exactly this class of incident in its own stack. A context-window routing error sent Sonnet 4 requests to servers configured for a different context length, affecting 16% of those requests at the worst hour and around 30% of Claude Code users at least once. The company found it through user reports and investigation, not through an alarm.
That is a lab with strong observability and a strong incentive to catch it early. A team running a homemade router on top of four providers should assume a harder version of the same problem. What to instrument is set out in the guide to monitoring agent drift in production.
Where this argument is weakest
The case for routing harder than I have argued
Three arguments cut the other way. Gartner predicts that by 2027 organisations will use small task-specific models at least three times more than general-purpose large models. Its stated reason is accuracy rather than cost. A general model answering a domain question is often worse, not just more expensive.
The contract extraction result supports that. A domain-trained small model beat five frontier models on precision while costing 78% to 97% less. If your task is narrow and you can fine-tune, the framing of this post is wrong: you are not trading quality for cost, you are buying both.
There is also a cost curve argument. Every figure here is a July 2026 list price. Prices have fallen steeply and repeatedly, and a routing threshold set today will be miscalibrated within two quarters.
What this post cannot settle
The honest limitation is that none of these benchmarks is your workload. LLMRouterBench, VAREX and RULER are public test sets, and public test sets carry contamination risk and distribution mismatch by construction. A model that has seen a benchmark during training scores above its real capability on novel data, and that inflation is not spread evenly across the models being compared.
I also disagree with the common advice to begin routing with a semantic classifier over free-text queries. Embeddings drift as your users change vocabulary, and the router degrades without firing an alert. Start with the task type you already know from your own call sites, because that signal does not drift.
The number I most want and cannot find is a published, independent measurement of quality loss from routing on a live enterprise workload, with a stated sample and time window. Every figure in this category is either a benchmark or a vendor claim. That gap is the reason to run your own eval suite before you trust anybody else's percentage.
A routing policy you can run this quarter
The sequence that survives contact with production is boring and it works. Classify your call sites by task type first, because you already know them. Route the two safest categories. Instrument before, not after.
| Signal | Why it matters | What a bad reading looks like |
|---|---|---|
| Cost per successful task | The only figure that nets retries, escalations and human rework against the saving | Cost per call falls while cost per completed task rises |
| Fallback and retry rate by route | Tells you whether the cheap tier is actually qualified for that traffic | Sustained retry rate above 10% on any single route |
| Per-route quality score | A blended score hides a bad route inside a good average | Aggregate score flat, one route trending down for weeks |
| Output tokens per task by tier | Cheap models often produce longer answers, eroding the price advantage | The cheap tier emitting more output tokens than the expensive one |
Then set the gate. A routing change ships only if cost per successful task falls and the per-route quality score holds. Both conditions, measured on the same window. That gate is what separates a saving from a deferred cost, and it is the same discipline described in the production readiness tests an AI pilot has to pass.
One structural choice is worth making early. If a task is high volume, narrow and stable, fine-tuning a small model beats routing to a cheap general one, and the VAREX result quantifies why. Whether to self-host that model is a separate decision, weighed in the comparison of open-weight models in production.
Frequently asked questions
What is LLM model routing?
Model routing sends each request to the cheapest model capable of handling it, rather than sending every request to the most capable model available. A classifier, heuristic or small model inspects the request and selects a tier. It differs from cascading, which runs the cheap model first and escalates on a failed confidence check, and from fallback, which only fires on error or timeout and cannot catch a wrong answer.
How much money does model routing actually save?
Published results range from 31.7% to 85% depending on the baseline. LLMRouterBench, covering 21 datasets and 33 models in January 2026, found up to 31.7% cost reduction while matching the best single model. RouteLLM reported 85% on MT Bench in 2024, but that was measured against using GPT-4 alone. Expect 30% to 50% in production once retries and escalations are counted.
Are small language models good enough for production?
For narrow, repetitive tasks, yes. Classification, flat-schema extraction and structured generation are handled well, and a fine-tuned small model can beat frontier models on a specific domain. They fail on repository-level code, long-context reasoning and any workflow with many sequential steps. Below 4 billion parameters the common failure is instruction following rather than capability, which fine-tuning largely fixes.
When should you not route to a cheaper model?
Avoid routing repository-level code generation, reasoning over long documents, and any agent workflow longer than about five sequential steps. Compounding is the reason. At 95% per-step accuracy a 10-step workflow succeeds only 59% of the time, against 90.4% at 99% accuracy. Also avoid routing extraction against nested JSON schemas until you validate the output shape, because schema echo can cost 45 to 65 accuracy points.
How do you measure quality loss from model routing?
Track cost per successful task rather than cost per call, and score quality per route rather than in aggregate. A blended score hides one degrading route inside a healthy average. Add a pre-merge gate on a representative eval set that blocks any routing change which drops quality below threshold. Silent regressions show up as a falling quality score with a zero error rate.
Where to start this week
Pull one week of production requests and tag each one by task type, not by prompt text. You will usually find that three or four call sites generate most of the volume, and that at least one of them is pure classification. That is your first route.
Then take the highest-volume route and run both tiers against 200 real requests from that route. Record cost per successful task for each, including retries. If the cheap tier does not win on that single number, the saving was never there, and you have found that out for the price of an afternoon rather than a quarter.
Related on the AI cost stack
Routing is one lever and it is not the largest. The others are set out in inference cost and gross margin, and the gate that keeps a saving honest is described in building an eval suite that catches regressions.
References
- Li, Zhang, Guo et al., LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing, arXiv:2601.07206, 12 January 2026. Used for the 31.7% cost reduction, the benchmark scale and the router accuracy on hard queries.
- LMSYS Org, RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing, 1 July 2024. Used for the 85%, 45% and 35% cost reductions and the GPT-4 call percentages.
- Barzelay, Azulai, Shapira et al., VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents, arXiv:2603.15118, March 2026. Used for schema echo, the 4B threshold and the fine-tuning gain.
- Lincoln, Whitehouse, Mar and Perera, A Few Good Clauses: Comparing LLMs vs Domain-Trained Small Language Models on Structured Contract Extraction, arXiv:2605.05532, 7 May 2026. Used for the F1 scores and the 78% to 97% cost reduction.
- Belcak, Heinrich, Diao et al., Small Language Models are the Future of Agentic AI, arXiv:2506.02153, June 2025. Used for the composition of agent workloads.
- Anthropic, A postmortem of three recent issues, September 2025. Used for the routing error figures.
- Gartner, Organizations will use small, task-specific AI models three times more than general-purpose LLMs by 2027, 9 April 2025. Used for the adoption forecast.
- TLDL, LLM API pricing table, updated 31 July 2026. Used for all list prices in this post.
The weakest part of this source base is the code figures. The HumanEval and SWE-bench Verified numbers come from a secondary compilation with no stated sample size or collection date, so they are directional only. The contract extraction paper evaluates a model built by its own authors. The per-task figures throughout use different benchmarks and different metrics and are not directly comparable to one another; the direction and rough size of each gap is the finding, not the point estimates. All prices are list prices dated 31 July 2026 and move frequently.
Related reading