From Sanskriti Khandelwal | Product & Market Analysis
Human in the Loop Is Not a Cop-Out. It Is the Architecture You Design First
On this page
Most agent deployments already have a human in the loop. Anthropic's own telemetry says 73% of agent tool calls across its API show human involvement of some kind. What almost none of them have is a loop anybody designed. The checkpoint is not an admission that the model is weak. It is the part of the system that decides who answers when the agent is wrong.
Key takeaways
- Oversight is already near universal and mostly accidental. 73% of agent tool calls on Anthropic's API show a human in the loop, while only 21% of organisations report a mature governance model for agentic AI.
- The approval prompt is the weakest checkpoint available. Claude Code users approve 93% of permission prompts. A gate that passes 93% of traffic is telemetry, not control.
- Reversibility beats importance as the design axis. Only 0.8% of agent actions in Anthropic's sample were irreversible, so a checkpoint aimed at undo cost touches a thin slice of traffic.
- Liability does not move to the vendor or the model. A Canadian tribunal held Air Canada responsible for what its chatbot told a customer and rejected the claim that the bot was a separate entity.
What human in the loop actually means when the system can act
Human in the loop means a person can inspect, change or stop an agent action before its effect lands. It is an architecture decision about where checkpoints sit and who holds override authority. It is not a confession that the model is weak, and treating it as one produces worse systems than treating it as a design surface.
The phrase covers three different designs. They are not interchangeable, and vendors use the same words for all three.
The three places a checkpoint can sit
Each position moves the cost somewhere else. None of them removes it.
| Design | Where the person sits | What it costs | Use it when |
|---|---|---|---|
| In the loop | Approves each qualifying action before it executes | Latency on every gated action | The action cannot be undone, or it is visible outside your company |
| On the loop | Watches a running agent and can interrupt it | Attention rather than latency | The work is reversible and the agent reports progress as it goes |
| Out of the loop | Reviews a sample of outcomes after the fact | Nothing at runtime, everything at incident time | Volume is high, blast radius is small, rollback is cheap and tested |
Most production systems run all three at once, on different action classes. A deployment using one position for everything has not made a decision. It has accepted a default.
Why the phrase became an insult
The stigma has a real origin. For two years, human in the loop was how vendors described a demo that needed help to finish. Buyers learned to hear it as a hedge, and sellers learned to stop saying it.
That reading is now backwards. The systems shipping with explicit checkpoints are the ones running in production, and the ones promising full autonomy are mostly still in pilots. The pattern behind those stalled pilots is documented in the breakdown of how agent pilots fail.
My position is blunt. A vendor who cannot describe their checkpoint design is a higher risk than one who admits their agent stops and asks. The first has not thought about failure. The second has.
The oversight already exists. Almost nobody designed it.
The best public measurement of what agents actually do comes from Anthropic's February 2026 research, which classified millions of tool calls across Claude Code sessions and public API traffic. Three of its numbers matter here.
80% of tool calls come from agents that appear to have at least one kind of safeguard. 73% appear to have a human in the loop in some way. And only 0.8% of actions appear to be irreversible, with sending an email to a customer given as the example.
Read those together and the shape of the problem changes. Oversight is not missing. It is spread evenly across a distribution of risk that is not even at all.
The governance surveys agree from the other side. Deloitte surveyed 3,235 technology and business leaders across 24 countries and found that only 21% have a mature governance model for agentic AI. The same work found 74% expect at least moderate agent use by 2027.
So three quarters of deployments have a person somewhere in the flow, and one in five has decided on purpose where that person belongs. The gap between those numbers is the subject of this post.
The approval prompt is the weakest checkpoint you can build
Start with the number that should end the debate about whether a prompt is a control.
Anthropic published in March 2026 that Claude Code users approve 93% of permission prompts. That figure was the stated motivation for building auto mode, which moves the approval decision to a model-based classifier instead of a person.
93% approval is fatigue, not judgement
A gate that passes 93% of what reaches it is doing something, but it is not filtering. It is logging, with a keystroke attached.
The decay is measurable and it tracks experience. In the same February 2026 research, users with fewer than 50 sessions ran full auto-approve about 20% of the time, and users past 750 sessions ran it more than 40% of the time. The people best placed to catch a bad action are the ones who have stopped looking.
This is not carelessness. It is what happens when you ask a person to make hundreds of low-information decisions a day. Design a checkpoint that fires constantly and you have designed one that gets ignored.
The law already names this failure
Regulators got there first. Article 14 of the EU AI Act requires that people assigned to oversee a high-risk system are enabled to remain aware of the possible tendency of automatically relying or over-relying on the output. The text names the phenomenon in brackets as automation bias.
The academic case is older and harder. Ben Green surveyed 41 policies requiring human oversight of government algorithms. His 2022 paper in Computer Law & Security Review concluded that they provide a false sense of security in adopting algorithms and enable vendors and agencies to shirk accountability. He proposed moving the burden from the individual reviewer to the institution that adopted the tool.
Green was writing about government, not commercial agents, so the transfer is an inference rather than a finding. It is a well-supported one. A reviewer with nominal authority and no real capacity to disagree is exactly what a 93% approval rate looks like from the inside.
Design the loop by reversibility, not by importance
The common instinct is to gate whatever feels important. That produces a long list, a slow agent and an exhausted reviewer. The better axis is narrower and it is testable.
Reversibility is the axis that matters
Ask one question about every action class your agent can take. Can we undo this in under five minutes without telling anyone outside the company? A yes moves the action out of the approval path entirely.
The point of the 0.8% figure is that this question sorts traffic very unevenly. Most of what an agent does is a read, a draft, a branch or a staged change. Very little of it leaves the building.
Rollback is the load-bearing assumption, so test it rather than assume it. If your rollback path has never been run under pressure, every action is effectively irreversible and your checkpoint budget is far larger than you think. The pre-production checks worth running first are in the list of tests a proof of concept has to pass.
Four checkpoints that are not a prompt
An approval dialogue is one instrument in a set of four. The other three are cheaper and they degrade more gracefully.
Preview and diff. Show the reviewer the change, not the intention. Anthropic's agent framework describes surfacing a real-time checklist of planned actions so users can adjust the plan while it runs, which is a different act from clicking yes on a description.
Budgets and rate limits. Cap spend, volume and reach so approval is not the only brake. A cap holds while the reviewer is asleep. A prompt does not.
Escalation the agent initiates. On the most complex tasks, Claude Code asks for clarification more than twice as often as humans interrupt it. Agents are better at knowing when they are stuck than humans are at spotting it from outside.
Sampled review with a named owner. Pick a percentage, write it down, and name the person who reads the sample. An unstated sample rate is not a control, and the same discipline governs watching behaviour change over time, covered in the piece on monitoring agent drift in production.
| Action class | Reversible? | Checkpoint | What the reviewer needs to see |
|---|---|---|---|
| Read internal data, search, summarise | Yes | Log only, sampled monthly | Nothing at runtime. Access scope at review time. |
| Write to a branch, draft a document | Yes, cheaply | Diff on merge, not on write | The change itself, beside what it replaced |
| Spend money, call a paid API at volume | Yes, at a cost | Hard budget cap plus alert | Cumulative spend against the cap, not each call |
| Message a customer, file a document, move funds | No | Approval before execution | Full payload, recipient, and the trigger behind it |
| Change permissions or create credentials | Technically yes, practically no | Approval plus a second reviewer | The identity granted, the scope, and the expiry |
The last row is the one teams add late. Agents that can grant themselves access sit outside the reversibility test entirely, which is why credentials get separate treatment in the analysis of non-human identity and agent access.
What the law already requires, and why it is a floor
For some systems the checkpoint is not optional. Article 14 of the EU AI Act requires high-risk systems to be designed so that natural persons can effectively oversee them while they are in use. The wording matters for buyers: the obligation lands at design time, on the provider.
The article then lists what the overseer must be able to do. Understand the system's limits. Interpret its output. Decide not to use it. Disregard, override or reverse the output, and interrupt the system through a stop button or a similar procedure. For biometric identification, Article 14(5) requires two competent people to verify an identification separately before any action follows from it.
European courts have narrowed what counts. In Case C-634/21 the Court of Justice held that an automated credit score playing a determining role in a lender's decision falls within the automated decision-making rules, because the score effectively settled the outcome. A reviewer who cannot realistically reach a different conclusion is not meaningful human intervention.
The United States is moving the same way with less certainty. Colorado repealed and replaced its AI Act in May 2026, delaying the effective date to 1 January 2027 and cutting the risk management duties. It kept a right to meaningful human review of adverse automated decisions, and it requires deployers to designate a trained individual with authority to override.
Treat all of this as a floor. These duties are drafted around consequential decisions about people. Your agent probably takes hundreds of actions a day that harm nobody's fundamental rights and could still cost you a customer. The disclosure side of the same regime is in the EU AI Act transparency checklist.
Liability does not move to the model
The clearest test of this predates the agent era and still governs the reasoning. In Moffatt v Air Canada, decided by the British Columbia Civil Resolution Tribunal in February 2024, an airline chatbot gave a passenger the wrong information about bereavement fares.
Air Canada argued that the chatbot was a separate legal entity responsible for its own actions. The tribunal rejected that outright, holding the company responsible for all information on its website whether it came from a static page or a chatbot, and found negligent misrepresentation.
That argument gets recycled in softer language every time an agent goes wrong. The vendor built it. The model produced it. None of it holds where a customer relied on what your system said.
Contracts help with cost and not with responsibility. A liability cap tells you how much of the bill your vendor absorbs, and the market terms are examined in the review of agent liability caps and remedies. It tells the customer nothing, and it tells a regulator nothing.
This is where the buyer-side framing flips. Demanding a designed checkpoint is not caution. It is the cheapest insurance available on a risk you cannot contract away.
What the loop costs, and the arithmetic most buyers skip
Review time is a line item and it belongs in the business case. Here is the arithmetic, with every assumption stated so you can substitute your own.
Take an agent handling 10,000 actions a month. Gate every action at 30 seconds of review and you have created 83 hours of monthly review work. That is half a full-time person, arriving as interruptions rather than as a job.
Now allocate by reversibility instead. Apply Anthropic's 0.8% irreversible share to the same volume and 80 actions need real review. Give each of those 3 minutes, because the reviewer reads a full payload rather than a dialogue box. Add a 5% sample of the remaining 9,920 at 30 seconds each.
| Regime | Actions reviewed | Time per action | Monthly review hours |
|---|---|---|---|
| Gate everything | 10,000 | 30 seconds | 83 hours |
| Gate the irreversible slice | 80 | 3 minutes | 4 hours |
| Sample the rest at 5% | 496 | 30 seconds | 4 hours |
| Designed loop, total | 576 | Mixed | 8 hours |
This is arithmetic on stated assumptions, not a measured result. The 0.8% share comes from Anthropic's cross-deployment sample and your own mix will differ. Replace the review times with a timed sample of your own team before quoting the total anywhere.
The ratio is the finding, not the hours. Same volume, same risk coverage, roughly a tenth of the labour, because the effort went where undo is impossible instead of where clicks are frequent.
Review labour also has to be netted off whatever the agent earns before any payback number means anything. Function by function figures are collected in the analysis of agent payback by business function.
Where this argument is weakest
Two objections deserve more room than a disclaimer.
The case for taking the human out
Anthropic's own response to its 93% approval rate was not to train better reviewers. It was to replace the reviewer with a classifier. Auto mode moved approval decisions to a model, and it became the default for new sessions on paid Claude Code plans in August 2026.
If a person catches little and costs much, a machine gate can genuinely be the safer design. Anthropic published the uncomfortable number alongside it: a 17% false-negative rate on real overeager actions, measured on 52 cases. The company calls that the honest number. It is also a small sample, and it is their own product.
Even so, the system falls back to manual approvals after three consecutive blocks or twenty across a session. The argument for automating the checkpoint still keeps a person at the end of it. That is the pattern, not an exception to it.
The part nobody can settle
There is no public dataset comparing incident rates between deployments with designed checkpoints and deployments without them. Every figure in this post measures prevalence or approval behaviour. None of them measures outcomes.
The strongest claim the evidence supports is narrow. Oversight is common, per-action approval decays predictably with use, and irreversible actions are rare. That checkpoint design lowers incident cost is a reasonable inference, not a measured result, including here.
There is a trust cost running the other way too. Reviewers asked to approve work they cannot evaluate stop trusting the tool and the process, a pattern visible in the gap between developer AI usage and developer trust.
Frequently asked questions
What does human in the loop mean in AI agents?
Human in the loop means a person can inspect, change or stop an agent action before its effect lands outside your systems. It is a design choice about where a checkpoint sits, not a statement about model quality. The three common placements are approval before each qualifying action, live monitoring with the power to interrupt, and sampled review after the fact. Most production deployments use all three at once, on different classes of action.
Is human in the loop required by law?
For some systems, yes. Article 14 of the EU AI Act requires high-risk AI systems to be designed so that natural persons can effectively oversee them. The overseer must be able to disregard, override or reverse an output and to stop the system. Colorado's revised AI law, effective 1 January 2027, preserves a right to meaningful human review of adverse automated decisions. Most commercial agents fall outside both, so the design question stays yours.
Does human in the loop slow AI agents down too much?
Only if you gate everything. Anthropic's telemetry found that 0.8% of agent actions were irreversible, so a checkpoint aimed at undo cost touches a thin slice of traffic. Gating every action at 30 seconds of review across 10,000 monthly actions costs about 83 hours. Gating the irreversible slice plus a 5% sample costs about 8 hours on the same assumptions. The cost comes from the placement, not the principle.
What is the difference between human in the loop and human on the loop?
In the loop means the agent waits for a person before the action executes, which adds latency to every gated step. On the loop means the agent proceeds while a person monitors it and can interrupt, which costs attention rather than latency. Use in the loop for actions you cannot undo. Use on the loop for reversible work where the agent reports what it is doing as it goes.
Why do approval prompts fail as a safety control?
Because approval rates climb until the prompt carries no information. Anthropic reports that Claude Code users approve 93% of permission prompts, and that users past 750 sessions run in full auto-approve more than 40% of the time, against roughly 20% for new users. Article 14 of the EU AI Act names this tendency directly and calls it automation bias. A gate that passes almost everything is telemetry rather than control.
Who is liable when an AI agent makes a mistake?
You are, in the first instance. In Moffatt v Air Canada the British Columbia Civil Resolution Tribunal rejected the airline's argument that its chatbot was a separate entity responsible for its own actions, and found negligent misrepresentation. Vendor contracts allocate cost between you and the vendor. They do not allocate responsibility between you and the person your agent harmed, which is a different question entirely.
Where to start this week
Open your agent's action log and add one column to it. Can this be undone in under five minutes without telling anyone outside the company. Sort by that column and you have your checkpoint map, usually in under an hour.
Then take the approval prompt your team clicks most often and change what it shows. Replace the description of the intended action with the actual payload or diff. If the reviewer still approves it every time, delete the prompt and move that effort to the row at the top of your sorted list.
Before you buy the agent
Ask the vendor two questions in writing. Which actions stop and wait for a person, and what does the reviewer see when they do. A vendor who answers with an autonomy percentage has not built the loop. Compare that against the failure modes that stall agent pilots.
References
- Anthropic, Measuring AI agent autonomy in practice, 18 February 2026. Source for the 73%, 80% and 0.8% shares, the auto-approve figures by session count, and the clarification ratio.
- Anthropic, How we built Claude Code auto mode, 25 March 2026. Source for the 93% approval rate, the 17% false-negative rate on 52 cases, and the fallback rules.
- European Union, AI Act Article 14, Human oversight. Source for the design-time duty, the automation bias wording, the override and stop requirements, and the two-person biometric rule.
- Deloitte, Agentic AI is scaling faster than guardrails, State of AI in the Enterprise 2026. Source for the 21% governance figure and the 3,235 respondent sample.
- Ben Green, The Flaws of Policies Requiring Human Oversight of Government Algorithms, Computer Law & Security Review vol 45, 2022. Source for the survey of 41 policies and the false-security argument.
- American Bar Association, BC Tribunal confirms companies remain liable for information provided by AI chatbot, February 2024. Source for the Moffatt v Air Canada holding.
- Skadden, Colorado repeals and replaces its AI Act, June 2026. Source for the delayed effective date and the surviving human review right.
The weakest thing about this source base: the three headline behavioural figures all come from one vendor's telemetry on one family of products. They are the best public measurement available and they are not an industry benchmark. Treat the 0.8% irreversible share as directional until you have measured your own.
Related reading