From Sanskriti Khandelwal | Product & Market Analysis
Build the LLM Evaluation Suite That Outlives Every Model You Ship On
On this page
Anthropic retired 9 Claude models in the 10 months to 5 August 2026. Its last four retirement notices ran 60 to 62 days. If you cannot decide inside that window whether the replacement is better or worse on your own work, you do not have an LLM evaluation gap, you have a release process running on hope. The model is rented. The eval suite is the part you own.
Key takeaways
- Model retirement notice has compressed to the policy floor. Anthropic's last four deprecation announcements gave 60, 60, 62 and 61 days before retirement, against a stated minimum of 60 days for publicly released models.
- Behaviour changes without the model string changing. Chen, Zaharia and Zou measured GPT-4 falling from 84% to 51% accuracy on the same prime number task between the March and June 2023 versions.
- Public benchmarks cannot serve as your release gate. NeurIPS 2025 research found task and reward design flaws distorting agentic benchmark results by up to 100% in relative terms.
- Twenty labelled failures beat 200 invented test cases. The binding constraint on an eval suite is whether its cases came from real traffic, not how many cases it holds.
The short answer
An LLM eval suite is a fixed set of task inputs, each paired with a grader that returns a score without a human in the loop. Build it from 20 real failures, grade with code where you can and a rubric where you cannot, and run it on every model, prompt and tool change.
Why the suite outlives the model you built it for
Every artefact in an AI product has a half life. The prompt gets rewritten. The retrieval index gets rebuilt. The model gets retired by a vendor on a schedule you do not control.
The eval suite is the exception. It encodes what "correct" means for your users, and that definition changes on your timeline, not the vendor's.
Both major labs publish their retirement calendars, and the calendars are dense.
Anthropic's deprecation page lists 9 model retirements between 28 October 2025 and 5 August 2026. Claude Sonnet 4 and Claude Opus 4, both released in May 2025, were retired on 15 June 2026, roughly 13 months after release. Claude Opus 4.1 lasted 12 months.
OpenAI runs the same pattern on a longer notice period. Its deprecations page commits to at least 6 months for generally available models and about 2 weeks for previews. The original GPT-5 model IDs, released in August 2025, were announced for shutdown on 11 December 2026.
Six months of notice is not six months of slack
Notice periods measure vendor courtesy. They do not measure the work.
The work is establishing whether the replacement model does your job as well as the outgoing one. Without a suite, that assessment is a week of engineers reading outputs and disagreeing, repeated at every migration.
Anthropic's own documentation says as much. It advises customers to test their applications with new models "well before the retirement date". That instruction assumes you have something to test with.
| Model retired | Announced | Retired | Notice |
|---|---|---|---|
| Claude Sonnet 3.5 (two builds) | 13 Aug 2025 | 28 Oct 2025 | 76 days |
| Claude Opus 3 | 30 Jun 2025 | 5 Jan 2026 | 189 days |
| Claude Sonnet 3.7 | 28 Oct 2025 | 19 Feb 2026 | 114 days |
| Claude Haiku 3.5 | 19 Dec 2025 | 19 Feb 2026 | 62 days |
| Claude Haiku 3 | 19 Feb 2026 | 20 Apr 2026 | 60 days |
| Claude Sonnet 4 and Claude Opus 4 | 14 Apr 2026 | 15 Jun 2026 | 62 days |
| Claude Opus 4.1 | 5 Jun 2026 | 5 Aug 2026 | 61 days |
Dates are taken from Anthropic's published deprecation history, retrieved 23 August 2026. Notice is calculated as calendar days between the announcement date and the retirement date. Partner platforms such as Amazon Bedrock and Google Cloud set their own schedules, which can differ.
What an eval actually is, and three things it is not
The word has been stretched until it means nothing. Fix the definition before you build anything.
The test case is a fixed input: a document, a query, a conversation prefix, a tool state. The grader is a function that takes the model output and returns pass, fail or a score.
Both halves have to be pinned. A case with no grader is a demo. A grader with no fixed case is a vibe.
Three things that are not evals
A dashboard of latency, token spend and error rates is observability. It tells you the system is running, not that it is right. Cost visibility matters, and it is a separate discipline covered in the piece on why cheaper tokens produce bigger bills.
A thumbs up and thumbs down widget is feedback. Response rates are low, self selected, and skewed toward the angry. Useful as a source of test cases, useless as a gate.
A leaderboard score is a benchmark. It measures a population of tasks that is not your population of tasks.
LangChain's State of Agent Engineering report, based on 1,340 responses collected in late 2025, found 89% of organisations had implemented some form of observability while 52.4% ran offline evaluations. That is vendor research with a commercial interest in the answer, so treat it as directional. The direction is that watching is far more common than grading.
Start with 20 failures, not 200 test cases
The most common way an eval project dies is starting with the taxonomy. Someone opens a document, writes eight quality dimensions, and nobody ever grades anything.
Invert it. The criteria fall out of the failures, and they do not exist before you look.
Twenty real failures beat 200 synthetic cases
Pull 20 outputs your users complained about, or that you would be embarrassed to ship. Write one sentence per output explaining what was wrong. Do not generalise yet.
Then group those 20 sentences. You will typically land on 3 to 5 clusters: it invented a fact, it ignored an instruction, it used the wrong tone. The other two are shape and tool choice: it called the wrong tool, or returned the right answer in an unparseable shape.
Those clusters are your criteria, grounded in observed damage rather than in an idea of quality.
Expect the definition to move while you work. Shankar and colleagues at Berkeley documented this precisely in their UIST 2024 study of LLM-assisted evaluation. They name it criteria drift: users need criteria to grade outputs, but grading outputs is what teaches them the criteria.
Their study also found participants reinterpreting existing criteria as they graded more, based on interviews with nine industry practitioners. That is a small sample and the paper says so.
When a criterion changes meaning, bump the suite version and re-baseline. Scores across a criterion change are not comparable, and treating them as comparable is how teams convince themselves a regression is an improvement.
The three grader types, and when each one is honest
Graders are not interchangeable. Each is honest about a different class of failure, and the wrong one measures nothing.
Deterministic checks first
Code that returns a boolean. Does the output parse as valid JSON, contain the required field, stay under the token limit, avoid the banned string, match the expected tool call.
These are cheap, fast and exactly reproducible. They should carry the majority of your cases. Most teams underuse them because format failures feel unglamorous, and format failures are what break production.
Rubric graders second
A model grades the output against a written rubric. This is the only practical option for open-ended qualities such as faithfulness to a source document or whether a refusal was appropriate.
It is also the layer that quietly fails. See the next section before you build your suite on it.
Human review, kept deliberately small
Humans are the calibration set, not the pipeline. Keep 20 to 40 cases that a person grades on a schedule, and use them for one purpose: measuring whether your rubric grader still agrees with your team.
If you cannot afford human labels at all, you cannot validate your automated graders, and you should say so out loud.
| Grader | Honest about | Cost per run | Where it fails |
|---|---|---|---|
| Deterministic code | Structure, schema, tool selection, hard constraints | Effectively zero | Cannot judge anything open ended, and passes fluent nonsense |
| Rubric grader (model as judge) | Faithfulness, tone, instruction following | Model spend, scales with suite size | Inherits the failure modes of the model being graded |
| Human review | Whether the rubric grader is still calibrated | Highest, and it is people time | Slow, inconsistent between raters, and it does not scale |
Where LLM judges break, and how to find out
Using a model to grade a model is the practical default, and it is the weakest link in most suites. The failure is rarely that the judge is bad. It is that nobody ever measured whether it was good.
The proxy assumption is the one nobody tests
Chehbouni and colleagues published a position paper in August 2025 arguing that adoption of LLM judges has outpaced scrutiny of their validity. They isolate four assumptions that get made silently: that judges proxy human judgment, that they evaluate competently, that they scale, and that they are cheap.
That paper is a position piece, not an empirical study, and it presents no measurements of its own. I cite it for the framing, not for evidence. Each of those four assumptions is testable on your own data, and most teams have tested none of them.
Reported agreement rates between LLM judges and human annotators do exist in the literature, and they vary widely by task. A judge validated on summarisation tells you nothing about a judge grading tool selection in an agent loop.
If the generator and the judge are the same model family, correlated errors are the default. A model that is confidently wrong about a domain will grade a confidently wrong answer as correct.
Two cheap mitigations. Use a different model family for the judge than for the generation. And include known-bad outputs in the suite as negative controls, so a judge that has stopped discriminating shows up as a suite that suddenly passes everything.
The trust question underneath this is not only technical. Developers already discount AI output more than usage figures suggest, a gap examined in the piece on developer trust running behind developer usage.
Why public benchmarks cannot be your release gate
Benchmarks are useful for one thing: shortlisting which two or three models to run your own suite against. Beyond that they are actively misleading, and the 2025 and 2026 literature on this is unusually direct.
The failures are in task and reward design, not just contamination
Research presented at NeurIPS 2025 audited widely used agentic benchmarks and found flaws in task setup and reward design. SWE-bench Verified used insufficient test cases. Tau-bench counted empty responses as successful.
The authors state that such issues can under or overestimate agent performance by up to 100% in relative terms. Applying their checklist to CVE-Bench cut the overestimation there by 33%.
This is not a claim that models are worse than reported. It is a claim that the measurement instrument has an error bar wide enough to swallow the difference between any two models you are choosing between. The same gap between benchmark scores and repository reality is the subject of the analysis of what coding agents do in real repositories.
An audit of Chatbot Arena published in April 2025, The Leaderboard Illusion, found practices that break the assumptions underneath the ranking. The authors identified 27 private model variants tested by one provider ahead of a single release. They reported 205 models removed from the arena without public notice against 47 officially deprecated.
They also found up to 26.5% of prompts were duplicates or near duplicates, and that submitting two identical checkpoints under different aliases produced scores 17 points apart. The arena's operators published a rebuttal disputing the framing. Either way, a scoreboard where identical models score differently is not a gate.
Which scoreboard you look at changes the answer entirely, a point developed in the piece on the three separate scoreboards for AI coding tools.
Where this argument is weakest
The case for evals is easy to overstate, and two parts of it are softer than the confident version admits.
The GPT-4 result from Chen, Zaharia and Zou is the most cited evidence that hosted models change underneath you. It is a real measurement. Its most striking task was criticised after publication because the prime number set contained only primes. A model that answered "prime" to everything scored well in March and poorly in June, without its reasoning necessarily degrading.
That criticism does not overturn the paper, which found shifts across several tasks including code generation. It does mean the headline figure is weaker evidence for capability loss than it is usually presented as being. Cite it for the existence of drift, not for its magnitude.
A suite is a snapshot of what mattered when you wrote it. Optimise against it long enough and you get a system that scores well and serves users worse, which is the same gaming dynamic the benchmark critics describe, reproduced inside your own company.
I would rather have that problem than the alternative. But it is a real cost, and the mitigation is boring: retire cases that no longer discriminate between models, and add cases from live failures every month. A suite that has not changed in six months has stopped measuring your product.
For a small team shipping one narrow feature, a formal eval suite can be over-engineering. If 30 minutes of reading outputs answers the question, read the outputs. The suite earns its keep when the same question gets asked repeatedly by different people.
Making it a release gate rather than a report
A suite that runs when someone remembers is a document. A suite wired into the deploy is a control.
Absolute scores are close to meaningless because they depend on case difficulty. Deltas against the last known-good run are meaningful.
Set the rule so it can fail. A deterministic check that regresses at all blocks the release. A rubric score that drops more than an agreed threshold blocks the release. If no configuration of results can block a release, you have built reporting and called it a gate.
| Weak setup | What it produces | Setup that works |
|---|---|---|
| Cases written by the team, from imagination | High pass rates and unchanged production failures | Cases pulled from logged failures and complaints |
| Judge model never validated against humans | A number nobody can challenge or defend | 20 to 40 human-labelled cases checked on a schedule |
| Suite runs quarterly, or before big launches | Regressions found after the migration deadline | Runs on every model, prompt, tool and index change |
| Report emailed to the team | Discussion, then a ship decision made on instinct | Pipeline step that can fail the build on a delta |
| Same model family generates and grades | Correlated errors and inflated pass rates | Different family for the judge, plus negative controls |
Rubric graders cost model calls, and a suite that runs on every commit multiplies that by your merge rate. This is the point where teams quietly reduce the run frequency and lose the gate.
Size it deliberately. Deterministic checks on every commit, the full suite including rubric graders on every merge to main and every model change. If that is unaffordable, cut the number of rubric cases rather than the frequency, because frequency is what makes it a gate.
The same buy-side question applies to eval tooling itself. Most of what a first suite needs is a test runner and a results table, which argues for building it, on the reasoning set out in the build versus buy analysis for coding agents. Buy when you need collaborative labelling and trace-linked review, not before.
Frequently asked questions
What are LLM evals?
An LLM eval is a test case paired with a grader. The test case is a fixed input, and the grader returns a pass, a fail or a score without a human reading it. A suite is a collection of those cases covering the failures you actually care about. Evals differ from benchmarks because the cases come from your own traffic, not a public dataset.
How do you build an eval framework from scratch?
Start by collecting 20 real failures from logs or user complaints. Write down why each output was wrong in one sentence, then group those sentences into 3 to 5 criteria. Turn each criterion into a grader: code where the check is deterministic, a rubric prompt where it is not. Run the suite on your current model to get a baseline. Everything after that is expansion, not design.
How many test cases does an LLM eval suite need?
There is no fixed number, and coverage matters more than count. A suite of 30 to 60 cases that map to distinct failure modes is more useful than 500 near duplicates. Add a case when a new failure reaches production, and stop adding cases that never change verdict between models. The suite should be small enough that you run it on every change.
Is LLM-as-a-judge reliable?
It depends on the task, and that is the problem. Judges are usually validated once against human labels on one task, then reused everywhere. A 2025 position paper argues the four core assumptions behind LLM judges, that they proxy human judgment, evaluate competently, scale, and cost little, have all outpaced the evidence. Measure your judge against human labels on your own data before trusting it.
Why can't you just use public benchmarks like MMLU or SWE-bench?
Public benchmarks measure a different population than your users. Research published at NeurIPS 2025 found task and reward design flaws in widely used agentic benchmarks. SWE-bench Verified used insufficient test cases, and tau-bench scored empty responses as successes. Those flaws distort reported performance by up to 100% in relative terms. Benchmarks are useful for shortlisting models and useless as a release gate.
How often should you run evals?
On every change that can alter output: model version, prompt, retrieval index, tool definition or temperature setting. Also run on a schedule, because hosted models change underneath a stable model string. Vendor retirement notices now run about 60 days at Anthropic, so a suite that only runs quarterly will discover a regression after the migration deadline rather than before it.
Where to start this week
Two hours, and you will have something that already works better than nothing.
Open your logs and pull the 20 worst outputs from the last month. Write one sentence per output on what was wrong. That list of sentences is your first criteria set.
Then take the 5 that are checkable in code, write those assertions, and run them against your current model to record a baseline. Put the baseline in version control with a date on it. The next time a vendor sends a 60 day retirement notice, that file is the difference between a migration and a gamble.
Related on measurement
Evals answer whether a model works for you. For why so many deployments never get that far, read the breakdown of how agent pilots actually fail. For why the published scores disagree with each other, read the three scoreboards piece.
References
- Anthropic, Model deprecations, retrieved 23 August 2026. Used for all retirement dates, notice periods and the 60 day policy floor.
- OpenAI, Deprecations, retrieved 23 August 2026. Used for the 6 month, 3 month and 2 week notice policy and the GPT-5 shutdown date.
- Chen, Zaharia and Zou, How is ChatGPT's behavior changing over time?, arXiv 2307.09009, July 2023, revised October 2023. Used for the 84% to 51% figure.
- Zhu et al., Establishing Best Practices for Building Rigorous Agentic Benchmarks, arXiv 2507.02825, NeurIPS 2025 Datasets and Benchmarks track. Used for the SWE-bench Verified and tau-bench flaws, the 100% relative misestimation and the CVE-Bench figure.
- Singh et al., The Leaderboard Illusion, arXiv 2504.20879, April 2025. Used for the 27 private variants, 205 versus 47 removals, 26.5% duplicate prompts and 17 point checkpoint spread.
- Shankar et al., Who Validates the Validators?, UIST 2024, arXiv 2404.12272. Used for criteria drift and the nine practitioner interviews.
- Chehbouni et al., Neither Valid nor Reliable? Investigating the Use of LLMs as Judges, arXiv 2508.18076, August 2025. Used for the four assumptions framing.
- LangChain, State of Agent Engineering, published 12 June 2026. Survey of 1,340 respondents, fielded 18 November to 2 December 2025. Used for observability and evaluation adoption rates.
The weakest source here is the LangChain survey. It is vendor research with a commercial interest in the finding, its respondent base skews 63% technology sector, and it is self selected. It is reported with its sample size for that reason and no argument in this post rests on it. The GPT-4 drift figure is also weaker than its popularity implies, for the reason stated in the limitations section. Retirement dates were correct on 23 August 2026 and vendors revise them.
Related reading