From Sanskriti Khandelwal | Product & Market Analysis
Data Lineage, Permissions and Freshness: Three Checks Before an Agent Acts
On this page
Outdated documents sitting in a retrieval index reduce answer accuracy by at least 20% across mainstream language models, measured on a benchmark published at ACL 2025. That is not a model defect and a stronger model does not repair it. Data lineage, permissions and freshness are the three properties an agent needs verified at the moment it acts, and most organisations still treat all three as a certificate they earned last quarter.
Key takeaways
- Outdated information in the retrieved context costs at least 20% of answer accuracy. On the HoH benchmark published at ACL 2025, the strongest model tested scored 51.7% when superseded facts sat beside current ones, and smaller models fell to 40.0% and 29.9%.
- The average data team was already absorbing 67 incidents a month before any agent arrived. A poll of 200 data professionals found 15 hours to resolve the average incident, and 74% said business stakeholders noticed the problem first.
- Lineage, permissions and freshness fail in different ways, so a single readiness score hides the thing you need. A gate that returns three separate verdicts tells you which one to repair. A percentage does not.
- Gartner expects the share of AI budget spent on data readiness to rise sevenfold between 2025 and 2029. That is a forecast rather than a measurement, and it signals that this work is now priced as infrastructure rather than as cleanup.
The short answer
Data lineage for AI agents is the ability to trace, at request time, which upstream sources produced the value an agent is about to use. It matters because agents act without a human reading the number first. Paired with a permission check and a freshness check, it is the smallest gate that catches the three failures that actually stop deployments.
AI-ready data is not a project you finish
Most data readiness programmes are shaped like a certification. A team audits the estate, repairs what it finds, publishes a policy, and declares the warehouse fit for AI.
That shape does not survive contact with a system that reads and writes continuously. An agent reads a field, decides something, writes a record, then reads that record back a few seconds later as the input to its next step.
So the useful question is not whether your data is AI-ready in general. The question is whether the agent can establish, at the instant it is about to act, that this specific field is traceable, permitted and current. Those are three separate lookups against three different systems, and hardly anyone has wired them into one call.
What actually changed when agents arrived
Dashboards tolerated unexplained and slightly stale data because a person sat between the number and the decision.
Removing that person does not simply remove a review step. It removes the only component in the system that was holding the three properties together.
My position is that the industry mislabelled this problem. It is described as a data quality problem, which sends teams to profiling tools and cleansing projects. It is closer to an observability problem, and the fix lives in the request path rather than in the warehouse. The sequenced repair work still matters, and the four layer remediation order for AI-ready data covers that side properly. This piece is about what has to keep running afterwards.
What the measurements actually say
Three numbers carry this argument, and they are unequal in strength. One is an experiment, one is a survey, and one is a forecast.
The experiment is the strongest. Researchers built HoH, a benchmark that constructs question sets in which a fact has genuinely changed over time, then places both the superseded version and the current version in the retrieved context. The paper was accepted at ACL 2025, which puts it through peer review rather than leaving it as a preprint.
The results are worth reading closely. The strongest model tested scored 51.7%, while smaller models reached 40.0% and 29.9%. The authors report a drop of at least 20% across mainstream models, and note that some configurations perform worse than chance. The failure mode is not hedging. Models answer confidently from the outdated document.
The incident numbers are older than they look
The most quoted operational figures in this field come from a Wakefield Research poll commissioned by Monte Carlo. They circulate constantly with a current year attached to them. They are from a poll of 200 data professionals conducted in March 2023, and the comparison year is 2022.
That does not make them useless. It makes them a baseline from the period immediately before agents entered production, which is arguably the more interesting reading. Whatever load agents have added since, they added it on top of this.
| Measure | 2022 | 2023 |
|---|---|---|
| Data incidents per month | 59 | 67 |
| Share reporting detection time of 4 hours or more | 62% | 68% |
| Average time to resolution | Not stated | 15 hours |
| Business stakeholders find the issue first, all or most of the time | 47% | 74% |
| Average share of revenue exposed to data quality issues | 26% | 31% |
Source: Wakefield Research for Monte Carlo, 200 respondents, March 2023. The 2022 comparison figures come from the same survey series. Self-reported by data professionals, so incident counts reflect what each team chose to log.
Read the fourth row twice. In one year, the share of teams whose users find the problem before the monitoring does climbed from under half to nearly three quarters. That is a detection failure, not a quality failure, and detection is exactly what a runtime check addresses.
Check one: lineage, asked at request time
Lineage has been sold for two decades as a diagram. You buy a catalogue, it crawls your warehouse, and it draws a graph a governance team reviews when a regulator asks.
An agent needs the same information in a different form. It needs a machine-readable answer, returned in milliseconds, to a narrow question about one field. Not a picture of the estate, and not a quarterly export.
Column level is the granularity that matters
Table-level lineage tells you that a report depends on a warehouse table. That is close to useless for an agent, because the agent is not acting on a table. It is acting on one value.
The OpenLineage column lineage facet is the clearest public specification of what the finer view should record. It separates two kinds of relationship. A direct transformation derives an output column from an input column through identity, transformation or aggregation. An indirect one records a column that shaped the result through a join, filter, group by or window clause.
That distinction is not academic. A field that shaped a WHERE clause never appears in the output, yet it can silently determine whether the row your agent is about to act on exists at all. Table-level lineage cannot represent that relationship. Column-level lineage with an indirect transformation type can.
The three questions lineage has to answer at runtime
Strip away the diagram and the useful interface is small. Given a field, the agent should be able to ask which upstream sources produced it, when the producing job last completed successfully, and whether any masking was applied on the way through.
The third question is the one teams forget. OpenLineage carries a masking flag on each transformation precisely because a value that has been obfuscated upstream looks like a normal value downstream. An agent reasoning over a masked field will produce a confident answer about data it never actually saw.
Platform vendors have converged on this shape. Snowflake describes Horizon Catalog as tracing AI-generated answers back to source through column-level lineage, and Databricks combines lineage, audit logs and inference traces in its account of governing agents through Unity Catalog. Convergence between two competing vendors is weak evidence about what works, and it is reasonable evidence about where the market believes the problem sits.
Check two: permissions, resolved per call
The permission failure is the one that ends careers rather than sprints. It also has the cleanest architectural answer, which makes the frequency of the mistake more frustrating.
On behalf of the user, not a service account
The default integration path is a service account with broad read access, because that is the fastest way to make a demo work. Every agent request then runs with the union of every permission anyone in the company holds.
Databricks describes the alternative directly, using on-behalf-of token passing so that the agent inherits the invoking user's permissions. Their formulation is worth quoting because it is the whole design in one sentence: if you cannot access a table, neither can the agent acting on your behalf. Snowflake makes the equivalent claim, that agents operate under the same access policies as humans.
The consequence people miss is that this makes permission a per-request property rather than a per-connector property. The same query from two different users must return different rows. Any caching layer that ignores identity quietly breaks the guarantee, and caching layers are added for latency by people who were not in the governance meeting. The identity side of this is examined further in the analysis of non-human identity and agent access.
The retrieval index is where permissions leak
Warehouse permissions are usually the part teams get right, because row and column policies already exist there. The leak happens in the retrieval index built over documents.
Embedding a document strips it from its access control list unless the pipeline deliberately carries the list through and filters on it at query time. A chunk of a compensation review sitting in a vector store has no owner, no group and no expiry. It has a similarity score.
I would treat any retrieval system that cannot filter by the requesting user's entitlements before the similarity search as unfit for internal data, regardless of what its accuracy benchmarks say. Filtering after retrieval is not equivalent, because the ranking has already been computed over documents the user cannot see. The practical differences between vendor approaches here are laid out in the comparison of enterprise assistant permission models.
Check three: freshness, the leg nobody instruments
Freshness is the check most often missing entirely, and the HoH result suggests it is the one carrying the largest measured cost. It is also the least served by existing tooling.
Age and staleness are different measurements
A document written four years ago is old. It is not stale if the policy it describes has not changed. A document written last month is fresh, and it is stale the moment a newer version supersedes it.
Retrieval systems have no access to that distinction. They rank on similarity, and a rescinded discount policy is textually almost identical to the current discount policy. A rescinded version that has sat in the index longer can rank above its replacement.
This is why deleting is an underrated engineering practice. The single highest return action available to most teams is not buying a freshness tool. It is removing superseded documents from the index, which nobody wants to own because deletion feels irreversible and archiving feels safe.
Four numbers worth writing down
Freshness becomes measurable once you stop treating it as one property. Four separate measurements cover the ground.
| Measurement | What it answers | Natural owner |
|---|---|---|
| Source age | How long since the underlying record was last written | Data platform team |
| Index lag | How long between a source change and the index reflecting it | Retrieval or platform engineering |
| Superseded rate | Share of retrieved documents that a newer version replaces | Content or knowledge owner |
| Corpus drift | Share of the whole index past its stated review date | Whoever signs off the corpus |
This is a measurement framework rather than a published benchmark. The four categories are our synthesis of how freshness is discussed in practice, and no industry baseline exists for any of them. Any thresholds you set will be local judgement.
Only the third number tells you about harm. The first two are plumbing metrics, and they will look healthy while the third is terrible. That is how a team ends up with green dashboards and a support agent quoting a price list that expired in March. The wider diagnosis of retrieval failures is worked through in the piece on diagnosing retrieval against long context.
Wiring the three checks into one gate
The three checks belong in one place, called once, returning three values. Splitting them across a governance dashboard, a security review and a retrieval config guarantees that nobody sees all three for the same request.
Give the gate a narrow contract. It receives a field or document identifier and a caller identity. It returns a provenance record or a null, a permission verdict, and an age with a supersession flag. It does not return a number between zero and one.
| Dimension | Certify once | Check at runtime |
|---|---|---|
| What it produces | A readiness statement about an estate | Three verdicts about one request |
| When it is wrong | Silently, from the day after the audit | Loudly, at the moment of the request |
| Cost shape | Large project, then nothing until the next audit | Small build, then permanent operating cost |
| Handles a schema change | No, until someone re-audits | Yes, if lineage collection is automated |
| Where it genuinely wins | Regulatory attestation, board reporting, and any obligation that requires a signature against a date | Nothing that needs a signature. A gate produces logs, not assurance. |
That last row is not a courtesy. If your obligation is a regulator asking what your controls were on a given date, a runtime gate answers the wrong question. You need both, and the mistake is running only the first.
Failure handling is where these systems are won or lost, and it is the part that almost never appears in a vendor demonstration. A permission failure must refuse outright, without a degraded fallback, because a partial answer assembled from whichever fragments the caller happens to be entitled to still discloses the shape of what they cannot see.
Lineage and freshness failures should degrade rather than refuse. If provenance is unknown, the agent can answer while declining to write, and should say which part it could not verify. If the source is stale, it can answer with the age stated in the response. Both preserve usefulness while removing the confident wrongness that the HoH result documents. The design space here overlaps with the architecture patterns for keeping a human in the loop.
Where this argument is weakest
Four objections deserve a hearing, and one of them is close to fatal for small deployments.
The first is latency. Three lookups in the request path add real milliseconds, and a catalogue was never designed as a low-latency service. Teams that add this naively will find their agent slower than the workflow it replaced, and they will remove the gate rather than cache it properly.
The second is that the HoH result measures question answering, not agent workflows. Extending a benchmark score to a multi-step system that also writes records is an inference on my part, not a finding of the paper. The direction is well supported. The magnitude in your setting is unknown.
The case for certifying once
The third objection is the strongest. For a small, stable, well-understood dataset, a one-time cleanup plus a nightly quality job genuinely is sufficient, and everything above is overhead. If your agent reads forty fields from one system that changes twice a year, build the cleanup and skip the gate.
The runtime argument earns its cost at the point where sources multiply, ownership fragments, and nobody can name every producer of a given field. That threshold arrives earlier than most teams expect.
The fourth objection is about evidence. The operational survey figures in this post are three years old and self-reported, the Gartner number is a forecast, and vendor product pages are marketing documents. Only the ACL result is a controlled measurement, and it tests one of the three legs. Anyone telling you the full three-check design is empirically validated is overstating what exists.
What this actually costs
Gartner's forecast, presented at its March 2026 Data and Analytics Summit, is that the share of AI spending directed at data readiness will rise sevenfold between 2025 and 2029. A forecast is a claim about belief rather than a measurement, and it signals budget expectations.
The concrete costs split three ways. Lineage collection is largely a build cost, because automated emitters exist and the work is instrumenting pipelines rather than hand-drawing graphs. Permission enforcement is mostly a rework cost, because it usually means undoing a service account integration that already shipped. Freshness is the only one that is a permanent operating cost, since somebody has to keep deciding what is superseded.
My advice on sequencing runs against the usual order. Do permissions first even though lineage is more interesting, because permission errors are the only ones in this set that are unrecoverable once they happen. Then do freshness, because it carries the largest measured accuracy cost. Lineage last, because it is the leg you can partly reconstruct after the fact and the other two you cannot. The broader budgeting question is worked through in the enterprise agent cost model.
Frequently asked questions
Why do AI agents need data lineage?
Because an agent acts on a value without a person reviewing it first. Lineage lets the agent establish which upstream sources produced the value, when the producing job last ran, and whether the field was masked along the way. Without that, an agent cannot distinguish a current figure from a stale one, cannot cite its source, and cannot be audited after it makes a wrong decision.
What is the difference between data lineage and data provenance?
Lineage describes the path data travelled through your systems, including the transformations applied at each hop. Provenance describes the origin and custody of a specific record, which is closer to a chain of evidence. In practice most tools use the terms interchangeably. For agent work the distinction that matters is granularity, because column-level detail is useful and table-level detail generally is not.
How do you control what data an AI agent can access?
Resolve permissions per request against the identity of the person who invoked the agent, rather than granting the agent a broad service account. Databricks and Snowflake both describe this pattern, where the agent inherits the caller's entitlements and cannot read anything the caller could not. Apply the same filter to any retrieval index before the similarity search runs, not after it returns results.
How fresh does data need to be for AI agents?
There is no universal threshold, because the answer depends on how fast the underlying fact changes and what the agent does with it. Measure four things instead: source age, index lag, the share of retrieved documents that a newer version supersedes, and the share of your corpus past its review date. The third number is the one that predicts harm.
Does column-level lineage matter for AI, or is table-level enough?
Column level matters, because an agent acts on a value rather than a table. The OpenLineage specification also records indirect relationships, where a column shaped a join, filter or grouping without appearing in the output. Those hidden dependencies determine whether the row an agent is reading exists at all, and table-level lineage has no way to represent them.
Is AI-ready data a one-time project or an ongoing process?
For a small dataset with few sources and slow change, a one-time cleanup plus a nightly quality job is genuinely enough. Beyond that it becomes ongoing, because lineage, permissions and freshness each drift independently between audits. The practical test is whether anyone in your organisation can name every producer of a given field. If not, certification will be out of date before it is published.
Where to start this week
Take the single highest-consequence write your agents can currently make, whether that is sending an external email, changing a price or updating a customer record. For the fields that decision reads, answer three questions by hand. Which job last wrote each one? Are the agent's credentials broader than the invoking user's? When was the source last confirmed current?
Doing that by hand for one workflow takes an afternoon and tells you which of the three legs is worst in your environment. Instrument that one first, and resist the temptation to buy a platform before you know which it is.
Related on this site
If you are earlier in the work, start with the four layer remediation order. If you are already in production, monitoring agent drift in production covers what happens after the gate is running.
References
- Ouyang, Pan, Cheng, Yan, Luo, Lin and Liu, HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generation, Proceedings of ACL 2025. Used for the 20% accuracy drop and the per-model scores.
- Monte Carlo and Wakefield Research, The Annual State of Data Quality Survey, 200 data professionals polled March 2023. Used for all incident, detection and resolution figures.
- Gartner, Data and Analytics Summit 2026 Orlando, Day 2 highlights, 10 March 2026. Used for the sevenfold data readiness spending forecast.
- OpenLineage, Column Lineage Dataset Facet specification. Used for the direct and indirect transformation types and the masking flag.
- Databricks, Governing AI agents at scale with Unity Catalog, May 2026. Used for on-behalf-of token passing and dual identity audit logging.
- Snowflake, Horizon Catalog product documentation. Used for column-level lineage tracing of AI-generated answers and agent access policy claims.
- Gartner, Lack of AI-Ready Data Puts AI Projects at Risk, 26 February 2025. Background on the AI-ready data definition.
The weakest thing about this source base: only reference 1 is a peer-reviewed controlled measurement, and it tests retrieval accuracy rather than agent workflows. Reference 2 is self-reported and three years old. Reference 3 is a forecast. References 5 and 6 are vendor product claims, cited for what those vendors have built rather than as evidence that the approach works. Gartner's newsroom blocks automated retrieval, so both Gartner figures were confirmed from indexed excerpts of those releases rather than from the page body, and should be re-verified before republication.
Related reading