From Sanskriti Khandelwal | Product & Market Analysis
Data Freshness Is the AI-Ready Data Layer Nobody Owns
On this page
Give a top model a retrieved document containing the wrong answer, and it abandons its own correct answer more than 60% of the time. That measured behaviour sits underneath a large share of agent errors filed as hallucination. The document was out of date, the agent believed it, and nothing in the governance programme was watching the clock.
Key takeaways
- Models defer to the document, even when the document is wrong. ClashEval tested six leading models on more than 1,200 questions across six domains and found they override their own correct prior knowledge over 60% of the time.
- Agents notice that a fact changed and still act on the old one. On the STALE benchmark the strongest model scored 92% at spotting a superseded belief and 30% at refusing a question built on it, for 55.2% overall.
- Freshness is already law for personal data. Article 5(1)(d) of the GDPR requires personal data to be accurate and, where necessary, kept up to date, with inaccurate data erased or rectified without delay.
- The budget is arriving before the practice. Gartner told its 2026 Data and Analytics Summit that the share of AI spending going to AI data readiness will rise sevenfold between 2025 and 2029.
Freshness is not data quality, and treating it as quality is why it gets skipped
Short answer
Data freshness is whether the record an AI system reads still describes the world at the moment it acts. Data quality asks whether a field is correct. Freshness asks whether it is still correct today. Agents fail on the second question because they retrieve by similarity, and a superseded document is extremely similar to the current one.
Data quality has a settled vocabulary. Accuracy, completeness, consistency, validity, uniqueness. Each of those is a property you can test against a field and then record.
Freshness is different in kind. It is a property of the gap between when a fact was true and when a system used it. You cannot certify it once and move on. You can only monitor it, and almost nobody is funded to.
That difference is why freshness falls through every governance programme built on quality thinking. A quality programme produces a scorecard. A freshness programme produces an alarm, and alarms need an owner and a rota.
Quality is a property of a field. Freshness is a property of a moment.
Consider a customer address. It is valid, complete and correctly formatted. It also stopped being where the customer lives 14 months ago. Every quality check passes and the shipment goes to the wrong city.
A human reading that record has a chance of catching it. They notice the account was opened in 2019, or that the region does not match the phone number. An agent dispatching a courier does not pause for that.
The same asymmetry runs through pricing, entitlements, policy documents and contract terms. The sequencing question of which layer to repair first is covered in the remediation order for AI-ready data. This piece is about the layer that sequence assumes somebody is already maintaining.
The four ways a fact goes stale
Not all staleness behaves the same way, and the remedies differ. Naming the four modes separately is the first useful thing a team can do, because three of them are invisible to conventional profiling.
| Failure mode | What it looks like | What the agent does |
|---|---|---|
| Superseded version | The old policy and the new policy are both indexed | Returns whichever scores higher on similarity, which is often the older, longer one |
| Expired fact | A rate, price or entitlement that was valid until a date | Quotes it with no awareness that the date has passed |
| Implicit invalidation | A newer record does not contradict the old one, it makes it irrelevant | Keeps acting on the old state because nothing looks like a conflict |
| Missing effective date | The document carries no timestamp at all | Cannot rank by recency even when instructed to prefer recent sources |
The four modes are our framing, not a published taxonomy. The third row maps to the implicit conflict case defined in the STALE benchmark, and the second to the temporal shift perturbation used in the Epi-Scale evaluation set.
The document wins the argument, and that is the whole problem
The standard defence against stale content is that the model knows better. It does not behave that way under test.
ClashEval, published in the NeurIPS 2024 Datasets and Benchmarks track, built a dataset of over 1,200 questions across six domains, including drug dosages, Olympic records and locations. The researchers then perturbed the answers inside the retrieved content, from subtle inaccuracies through to blatant errors.
Six leading models were benchmarked, including GPT-4o. The finding is blunt: models are susceptible to adopting incorrect retrieved content, overriding their own correct prior knowledge over 60% of the time.
Read that against how a retrieval stack is usually sold. The pitch is that grounding a model in your documents removes hallucination. What grounding actually does is transfer the failure from the model to the corpus, which is a real improvement only if somebody maintains the corpus.
Low confidence makes it worse, which is exactly backwards
ClashEval found two moderating patterns. The more unrealistic the wrong content, the less likely a model is to adopt it. And the less confident a model is in its own initial answer, measured through token probabilities, the more likely it is to take the document's word for it.
The second pattern is the dangerous one. Models are least confident on niche, internal, low-frequency facts. Those are precisely the facts your enterprise corpus exists to supply, and precisely where your corpus is most likely to hold a five-year-old copy.
So the deference is strongest where your data is weakest. That is not a model defect you can prompt around. It is an argument for spending on the corpus rather than on the model, and it is the clearest reason we would fund freshness before funding another evaluation of frontier models.
Agents cannot reliably tell when a fact has expired
A second body of work asks the sharper question. Not whether a model follows a wrong document, but whether it can notice that something it already believes has stopped being true.
The STALE benchmark, published as a preprint in May 2026, tests three things: whether a model can identify that a stored belief is stale, whether it rejects a question that presupposes the outdated fact, and whether it changes its behaviour without being told to.
The strongest model reached 55.2% overall, and most systems scored below 25%. The authors describe the pattern as a pervasive gap between retrieving updated evidence and acting on it.
Noticing the change is not the same as acting on it
The breakdown is more useful than the headline. On co-referential conflicts, where a new fact directly replaces an old one, the best model reached 92% at state resolution. On refusing a question built on the superseded fact, the same model reached 30%.
On propagated conflicts, where the change reaches a related fact rather than the one stated, state resolution fell to 69% and premise resistance to 14%. Behaviour adaptation landed at 55%.
That shape has an operational reading. Ask an agent directly whether a fact has changed and it will often say yes. Ask it to do something that quietly depends on the old fact and it will do it. Almost every production interaction is the second kind, which is why monitoring agents in production has to include the corpus, not only the model.
What has actually been shown to help
Two results, one from 2025 and one from 2026, point the same direction. Neither involves a better model. Both involve organising the evidence before the model sees it.
Researchers at the University of Tuebingen built two benchmarks for evolving knowledge: Temporal Wiki, from historical Wikipedia snapshots, and Unified Clark, from timestamped news accumulating over time. They then compared zero-shot answering, in-context learning, retrieval and a knowledge organisation step.
Organising the knowledge beats stuffing the window
On Unified Clark, with Llama 3.1 70B, in-context learning scored 0.43, retrieval scored 0.68 and the knowledge organisation step scored 0.76. On Temporal Wiki, restricted to questions the model first got wrong, retrieval reached 0.78 and organisation 0.81.
The consistent finding is that retrieval beats stuffing a long context, and that structuring the retrieved evidence beats plain retrieval. Long context windows did not make this go away, which is a separate argument developed in the piece on diagnosing retrieval before rewriting it.
A second 2026 preprint approaches the same problem from the reasoning side. It builds Epi-Scale, a 4,500-record set with 2,250 adversarial records carrying entity swaps, temporal shifts, logical contradictions and distractor evidence.
Standard retrieval scored a 63.0% macro average on the adversarial set, rising to 78.1% when the system was made to name the conflict before answering. On TruthfulQA with misconceptions injected into the retrieved context, the worst case moved from 15.0% to 62.0%.
Why dashboards tolerated staleness and agents do not
Stale data is not a new condition. Every reporting layer ever built has carried some. The difference is what sits at the end of the pipe.
A stale number in a dashboard meets a human who has context. They know the region was reorganised in March. They know the figure looks wrong. The number gets questioned before it becomes a decision.
A stale record read by an agent meets an API call. There is no pause, no eyebrow, no colleague to check with. This is the practical case for designing the human checkpoint deliberately rather than assuming one exists.
A stale number gets questioned. A stale action gets executed.
That single sentence is the reason freshness moved from a hygiene concern to a control. It also explains why the failure surfaces after the pilot rather than during it.
Pilots run on a curated corpus. Somebody assembled 200 documents, checked them, and pointed the agent at them. Production runs on the whole estate, which nobody has read since it was written.
Permissions have the same shape, and the two interact badly. An agent with access it should not have, reading a document nobody has reviewed since 2022, is the combination examined in the piece on enterprise search permissions gaps. Freshness and access scope are two axes of one control problem.
Freshness is already an obligation, not a best practice
Most teams treat data currency as good hygiene. For personal data in Europe it has been a legal requirement since 2018, and almost nobody enforces it as one.
Article 5(1)(d) of the GDPR states that personal data shall be accurate and, where necessary, kept up to date. It goes further: every reasonable step must be taken to ensure that inaccurate personal data is erased or rectified without delay. The UK regulator's guidance on the accuracy principle defines inaccurate as incorrect or misleading as to any matter of fact.
Read that alongside an agent that emails a customer their entitlement based on a superseded record. The accuracy principle is not a filing requirement. It is a live obligation about the data you act on.
The EU AI Act's Article 10 pushes in the same direction for high-risk systems, requiring data sets to be relevant, sufficiently representative and, to the best extent possible, free of errors and complete. It names updating explicitly among the data preparation operations that governance practices must cover.
One honest caveat, because it is load bearing. Article 10 addresses training, validation and testing data. It does not obviously reach the document store your retrieval agent queries at inference time. We would not build a compliance argument on it, and the broader duties are set out in the transparency checklist for the AI Act.
Build a freshness contract, not a cleanup project
The instinct after reading the above is to launch a data cleanup. We think that is the wrong move, and it is the one most likely to be running 18 months from now with nothing shipped.
A cleanup has no end state. The estate goes stale again while you are still working through it. What holds is a contract: for each class of data an agent reads, a stated maximum age, a named owner and a gate that blocks action when the age is exceeded.
| Data class | Staleness that matters | How it stays current | Gate before an agent may act |
|---|---|---|---|
| Pricing and entitlements | Hours to days | Read live from the system of record | No cached copy allowed in the retrieval index |
| Policy and compliance documents | Weeks | Named owner attests each quarter | Effective date and superseded-by field required |
| Customer state and contract terms | Minutes | Read through to the source at request time | Any write action forces a re-read first |
| Reference and training material | Months | Annual retire-or-refresh review | Retired copies deleted from the index, not archived in it |
The intervals above are our proposal based on the failure modes described earlier. They are not derived from a published study, and any team adopting them should expect to move them after the first quarter of incident data.
Three fields, added to every source an agent can read
Effective date, review date, and superseded-by. Three fields, and most retrieval stacks can filter and rank on them within a sprint.
Effective date lets the system prefer recency instead of guessing at it. Review date turns silence into a signal, because a document past its review date is a document nobody has vouched for. Superseded-by is the one that kills the first failure mode outright, by making the old version point at the new one instead of competing with it.
The fourth move is a test, not a field. Put an outdated document into a staging index and check whether the agent still answers correctly. That belongs in the same harness as the rest of your evaluation suite, and it is the only one of these four steps that tells you whether the other three worked.
Where this argument is weakest
Three problems with the case above, stated plainly, because two of them are large.
The benchmarks are not your corpus
ClashEval uses drug dosages and Olympic records. STALE uses synthesised user states. Epi-Scale draws on HotpotQA, Natural Questions and FEVER. None of them is a company wiki, a CRM or a contract repository.
These results establish that models defer to retrieved content and struggle with expiry. They do not establish the rate at which that happens on your data. Anyone quoting the 60% figure as a prediction about their own deployment, us included, is extending it past what was measured.
Two of the four studies here are also preprints rather than peer-reviewed work, and both are recent. The STALE and Epi-Scale results should be treated as directional until they have been reproduced.
Nobody has measured enterprise staleness properly
We looked for a credible measurement of how much enterprise content is superseded, duplicated or expired. What exists is vendor material: storage suppliers quoting shares of redundant and obsolete data, with methodology that is not published and sample frames that are their own customers.
So the size of the problem inside a typical company is genuinely unknown. That is a real hole in this argument. The mechanism is well evidenced and the prevalence is not, and we would rather say so than borrow a number from a storage vendor's landing page.
There is also a fair counter-case for doing very little. If your agents only read three systems of record, queried live, you have no staleness problem worth funding. Freshness work earns its budget when an agent reads documents, not rows.
Frequently asked questions
What does data freshness mean in AI?
Data freshness is whether a record still describes the world at the moment an AI system reads it. It differs from data quality, which asks whether a field is correct in isolation. A customer address can be valid, complete and correctly formatted while being 14 months out of date. Quality checks pass, the agent acts, and the result is wrong.
Why do AI agents give outdated answers?
Retrieval ranks documents by similarity, and a superseded version of a document is highly similar to the current one. If both are indexed, the older one can win. Research also shows models tend to follow retrieved content rather than their own knowledge, so once the wrong document is selected, the model rarely corrects it. The failure is in the corpus, not the model.
How often do language models follow a wrong retrieved document?
ClashEval, published in the NeurIPS 2024 Datasets and Benchmarks track, tested six leading models on more than 1,200 questions across six domains with deliberately perturbed source content. The models overrode their own correct prior knowledge more than 60% of the time. Deference rose when the model was less confident in its own answer, and fell when the injected error was extreme.
Is data freshness a legal requirement under GDPR?
For personal data, yes. Article 5(1)(d) requires personal data to be accurate and, where necessary, kept up to date, and requires every reasonable step to be taken so that inaccurate data is erased or rectified without delay. That obligation applies to data an AI system acts on, not only to data at rest. It does not extend to non-personal reference content.
How do you measure data freshness for an AI system?
Add three fields to every source an agent can read: effective date, review date and superseded-by. Then measure the share of retrieved documents past their review date, per query, in production. Pair that with a staging test where a known outdated document is injected and the agent's answer is checked. The first number tells you exposure, the second tells you consequence.
Does data lineage fix stale data?
Lineage makes staleness visible, which is necessary but not sufficient. Knowing that a figure came from a warehouse table refreshed six weeks ago lets you decide whether to trust it. It does not refresh the table, retire the superseded document or block the agent from acting. Lineage is the instrument. The freshness contract and its named owners are the control.
Where to start this week
Pick your single highest-volume agent workflow and list every source it reads. Not the systems, the sources: each index, each folder, each table.
Then run one query against that list. For each source, what is the date of the most recently reviewed document in it, and who reviewed it? Most teams cannot answer for more than half their sources, and that answer is the finding.
If you want a second afternoon's work, take one document you know was replaced, confirm the old version is still in the index, and ask the agent the question it answers. Record what comes back. That transcript will move a budget conversation further than any benchmark in this post.
Related on this site
This piece covers one layer. The order in which to repair the rest is set out in the AI-ready data remediation sequence, and the retrieval mechanics underneath it in diagnosing retrieval before you rewrite it.
References
- Wu et al., ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence, NeurIPS 2024 Datasets and Benchmarks Track. Used for the 60% override figure and the confidence pattern.
- Chao et al., STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?, arXiv preprint, 7 May 2026. Used for the 55.2% overall score and all task-level breakdowns.
- Ozer and Yildiz, Question Answering under Temporal Conflict, arXiv preprint, 8 June 2025. Used for the Temporal Wiki and Unified Clark accuracy figures.
- Chen et al., Does RAG Know When Retrieval Is Wrong?, arXiv preprint, 19 July 2026. Used for the Epi-Scale and TruthfulQA figures.
- Vu et al., FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation, Findings of ACL 2024. Background on model accuracy against fast-changing facts.
- European Union, GDPR Article 5, principles relating to processing of personal data. Used for the accuracy and currency obligation.
- European Union, AI Act Article 10, data and data governance. Used for the high-risk data set requirements.
- Gartner, Data and Analytics Summit 2026 Orlando, Day 2 highlights, 10 March 2026. Used for the AI data readiness spending figure.
The weakest thing about this source base: two of the four empirical studies are unreviewed preprints from 2026, and none of the four measured an enterprise corpus. The mechanism is well evidenced. The prevalence inside a given company is not, and no credible public measurement of it was found.
Related reading