From Ritu Raj | Product & Market Analysis
AI Red Teaming for Agents: Why an 81% Hijack Rate Changes Your Test Plan
On this page
NIST security researchers took the known attacks against AI agents and improved them. Task hijacking success went from 11% to 81% on the same benchmark, against the same model. Nothing about the agent changed, which is the whole case for AI red teaming as a standing programme rather than a launch-day checkbox. Most teams run it as neither.
Key takeaways
- Attacker skill, not model choice, moved the hijack rate from 11% to 81%. NIST's Center for AI Standards and Innovation reported that jump on the AgentDojo benchmark using attacks it built itself, against an unchanged agent.
- Testing each scenario once understates your exposure by roughly a quarter. The same NIST work found average attack success rising from 57% on a single attempt to 80% across 25 attempts per injection task.
- Agent red teaming tests actions, not answers. The unit of failure is a tool call made with real credentials, which is why prompt-only jailbreak suites tell you almost nothing about blast radius.
- Governance is running ahead of validation. In the 2026 SANS AI Survey, 76% of practitioners held a governance role for enterprise AI and more than half said no formal audit framework backed it.
What agent red teaming tests that model red teaming does not
Model red teaming asks whether a model will produce harmful text. Agent red teaming asks whether an attacker can make a system take an action it was never authorised to take, using its own tools, credentials and permissions. The output under test is a state change, not a paragraph.
That distinction stays academic until you price the failure. A jailbroken chatbot produces a bad answer that somebody has to read and act on. A hijacked agent files the payment, sends the email, drops the table or merges the branch on its own.
Microsoft's AI red team made a version of this point after 80 operations covering more than 100 generative AI products. Their third lesson is stated flatly: AI red teaming is not safety benchmarking. Benchmarks measure harms you have already named. Red teaming exists to find the ones you have not.
I would put it harder than they do. Most of what teams currently call agent red teaming is regression testing against a list of jailbreak prompts somebody found online, and it measures nothing a funded attacker cares about.
The tool call is the vulnerability, not the sentence
An agent's attack surface is the set of actions it can perform, multiplied by the set of places its instructions can arrive from. Indirect prompt injection is only the delivery mechanism. The tool is the payload, and the credential attached to that tool is the damage.
OWASP's Q1 2026 exploit round-up is a useful check on where real events actually land. Across 1 January to 11 April 2026 it catalogued eight incidents. They included remote code execution in Flowise, indirect prompt injection through Grafana, an agent deleting a user's inbox, privilege abuse inside a managed platform, and a vendor breach that reached training workflows. Only one of the eight carried a CVE.
That last detail should shape your programme design. If you are waiting for a vulnerability feed to tell you an agent is exposed, you will wait through most of the category. These failures come from configuration, design and supply chain rather than from a discrete code defect. The same logic drives the supply chain risk sitting inside MCP servers, where a dependency you never reviewed gains the right to act on your behalf.
| Dimension | Model red teaming | Agent red teaming |
|---|---|---|
| Unit of failure | A harmful or disallowed output | An unauthorised action taken with real credentials. |
| Attacker entry point | The user prompt | Any content the agent reads: email, tickets, web pages, files, tool responses. |
| Success criterion | The model said it | The system did it, and the audit log proves it. |
| Blast radius | Bounded by what a human does next | Bounded by the permission set attached to the agent. |
| Reproducibility | Reasonable, prompt in and text out | Poor, because environment state changes between runs. |
| Who can run it | A safety or ML team | Security engineering, with platform and application owners in the room. |
The last row is the one most organisations get wrong. Agent testing needs someone who can read an IAM policy, and safety teams usually cannot.
The number that should set your scope
In January 2025 NIST's Center for AI Standards and Innovation published agent hijacking evaluations built on AgentDojo, an open-source framework from ETH Zurich covering four environments: workspace, travel, Slack and banking. CAISI extended it with three attack categories it treated as priorities.
Those categories are worth copying directly into your own scope document. Remote code execution through agent tool use. Mass database exfiltration. Automated phishing sent through the agent's own communication tools.
Against agents powered by Anthropic's Claude 3.5 Sonnet, the strongest known baseline attack hijacked the agent 11% of the time, and NIST's strongest new attack succeeded 81% of the time. The model was the same in both runs. The prompt-injection defences were the same. Only the attacker improved.
Read that as a statement about attacker effort rather than about any one vendor. It is the distance between an opportunist reusing published techniques and an adversary willing to spend a fortnight on your specific agent. If your threat model includes the second person, a test suite built from the first person's material is not evidence of anything.
One attempt is the wrong test
The same NIST work varied the number of attempts rather than the technique. Average attack success rose from 57% on a single attempt to 80% when 25 attempts were allowed per injection task. Individual tasks varied widely around both figures.
This has a direct consequence for how you write test cases. A red team that runs each scenario once and records a pass is measuring luck. Attackers retry, and nothing in the agent's design makes the second attempt harder than the first.
Set a minimum attempt count per scenario in your methodology and publish it in the report. Twenty-five is a defensible starting point because a public evaluation used it. Anything lower needs a written reason.
Scope: the four surfaces worth testing
OWASP's agentic threats and mitigations taxonomy splits the problem into agent design, memory, planning and autonomy, tool use, and deployment and operations. That is a good analytical carve. It is a poor staffing carve, because no single owner exists for most of those boxes.
Four surfaces map better onto who actually holds the keys. Instruction and content, where injected text arrives. Tools and connectors, where actions execute. Memory and state, where an attack persists. Identity and permission, where damage is either bounded or not.
Start at the tool layer, not the prompt
Prompt testing is cheap, visible and easy to demonstrate to a steering committee. It is also the layer where mitigation is weakest and the least likely to change a decision. Every published defence against indirect injection is probabilistic, so a finding there tells you a number you cannot drive to zero.
The tool layer behaves differently. A finding that an agent will call a payment function with attacker-controlled arguments produces a fix you can actually ship: an allowlist, a typed schema, a confirmation step, a scoped token. Findings that produce deterministic fixes are worth more than findings that produce probability estimates.
Work backwards from the worst action in the tool manifest. Ask what the agent can do that you would never let a new contractor do unsupervised on day one, then design the test that forces exactly that call. This is the same reasoning that decides where a human approval step genuinely belongs rather than where it feels reassuring.
Memory and state are the surface nobody books time for
An injected instruction that lands in a shared vector store or a persistent scratchpad does not expire when the session ends. It waits. The next user, or the next agent, reads it as trusted context and acts on it.
Almost no test plan I have seen includes a cross-session case. The standard scenario runs one conversation, checks the output and tears the environment down, which is precisely the shape that hides persistence bugs. Write at least one scenario that plants content in session A and asserts on behaviour in session B, under a different user identity.
Identity is the fourth surface and the one that decides everything else. An agent operating under a broad service account converts a minor injection into a major incident. That is why non-human identity and access control belongs inside this programme rather than in a separate compliance workstream.
Cadence: what runs continuously and what runs quarterly
Most red teaming programmes die from cadence rather than from method. An annual exercise produces a document. A weekly one produces alert fatigue and a queue nobody triages. The workable pattern splits the work by cost and by trigger.
| Cadence | What runs | Who runs it | Output |
|---|---|---|---|
| Continuous, in CI | Automated injection and tool-abuse suite against every agent build | Application team, suite owned by security | Pass or fail gate, no report. |
| Monthly | Tool and connector inventory diff, permission review, new MCP servers | Platform and IAM | One page of changes and their blast radius. |
| Quarterly | Manual exercise against one production agent, full kill chain, 25 attempts per scenario | Internal red team or external firm | Findings with severity mapped to actions. |
| Event triggered | New tool with write access, new model version, new data source, incident | Whoever owns the change | Scoped test before the change ships. |
| Annual | Independent assessment and methodology review | External, different firm from quarterly | Assurance artefact for the board. |
Event triggers beat the calendar
Quarterly testing assumes the system is stable between tests. Agent systems are not. A new tool, a model upgrade or a new document source changes the attack surface more than three months of calendar time does.
So make the trigger structural. Any change that grants an agent a new write capability requires a scoped test before it ships, and the test is small because the change is small. This is the cheapest control in the whole programme and the one most often skipped.
Pair the cadence with production telemetry, because a test suite only sees what you thought to write down. Watching for behavioural drift in production agents is how you find the attack pattern your scenarios missed.
Staffing the programme without standing up a new team
You do not need a dedicated agent red team to start. You need one named owner, a written methodology and a budget line for one external exercise a year. Everything else can be borrowed from teams that already exist.
The scarce skill is not prompt creativity. It is the ability to read a tool manifest and an IAM policy in the same sitting and say what the worst reachable action is. That person usually sits in security engineering or platform, not in the AI team. The hiring signals look closer to what an agent ops engineer needs to know than to a traditional application pentester.
Buy the quarterly exercise externally for the first year. Internal teams test the system they built, which means they test the assumptions they already hold. An external firm brings different assumptions, and the point of red teaming is the assumption you did not know you had made.
A bug bounty is a channel, not a programme
Every major lab now runs a public safety or security bounty, and the model works. It is a reporting channel with an incentive attached, not a testing programme, and it covers the systems researchers can reach from outside.
Your internal agents are not reachable from outside. A bounty will never see the finance agent behind your VPN, so it cannot substitute for scheduled testing of the systems that hold your money. Treat the two as complements with different coverage, and do not let a bounty line item close out the testing budget.
Reporting: the three numbers an executive can act on
Red team reports fail at the last mile. A hundred-page findings document with severity ratings borrowed from web application testing gives an executive nothing to decide with. CVSS was never designed for a system whose worst outcome depends on a token scope.
Report three numbers per agent, every quarter, in the same format. What the agent can do at maximum, how often the team could make it do that, and how long the fix took. Everything else is an appendix.
| Number | Definition | What a bad reading looks like |
|---|---|---|
| Maximum blast radius | The worst action reachable with the agent's current permissions, named as a business outcome. | Anything involving money movement, customer data export or production write access. |
| Hijack rate | Share of scenarios where the team induced an unauthorised action, at a stated attempt count. | Above zero on any scenario whose blast radius is in the row above. |
| Time to remediate | Median days from finding to shipped fix, split by fix type. | Most fixes are prompt changes, which means nothing structural moved. |
The third row is the honest one. A programme whose remediations are mostly prompt edits has found real problems and applied cosmetic fixes.
Track those three against a maturity model rather than against a target. Where an organisation sits on the agent governance maturity ladder tells you which of the three numbers is even measurable yet. Pretending to report a number you cannot compute is worse than reporting a gap.
Where this argument is weakest
Three problems sit inside everything above, and a programme built without acknowledging them will oversell what it produces.
Coverage is the number nobody can report
Traditional security testing can state coverage against an asset inventory. Agent red teaming cannot, because the input space is unbounded and the failure modes are emergent. A report that says 47 scenarios passed is not saying that scenario 48 would have.
This is the same measurement gap that makes an internal eval suite useful and insufficient at once. Both tell you about the cases you imagined. Neither bounds the cases you did not, and any vendor claiming otherwise is selling a coverage metric it cannot define.
The 81% is one model, one framework, one date
The NIST result is the strongest public evidence in this post and it needs stating carefully. It used AgentDojo, one benchmark with four synthetic environments, against agents built on a single model version from October 2024, published in January 2025.
It is not a measurement of your agent. Defences have improved since, and a benchmark environment is far simpler than a production system with real authorisation checks. What transfers is the shape of the finding, which is that attack quality dominates, and that shape is unlikely to have reversed.
The second caveat is about the regulation people cite in this area. The EU AI Act requires documented adversarial testing under Article 55, but that obligation lands on providers of general-purpose models with systemic risk. If you deploy agents built on someone else's model, that article is not your compliance trigger, and a consultant telling you otherwise has not read it.
A 90-day build for a team that has none
Assume one part-time owner and no budget beyond a single external engagement. This sequence is deliberately ordered so that each step produces something the next step needs.
Days 1 to 30. Inventory every agent in production and every tool each one can call. Record the identity it runs as and the scopes attached. Most teams discover during this step that the inventory does not exist, and that finding is worth more than any test result.
Days 31 to 60. Write the methodology document. Scope, attempt count, the three attack categories from the NIST work, the four surfaces, severity defined by blast radius rather than by CVSS. Build the automated suite for the first surface and wire it into CI as a warning, not a gate.
Days 61 to 90. Run one manual exercise against the highest-permission agent you own. Flip the CI suite from warning to gate. Publish the first three-number report even if two of the numbers are gaps, because the gap is the baseline. If your agents are still in pilot, the same discipline applies through the readiness tests that decide whether a pilot ships at all.
Frequently asked questions
What is AI red teaming?
AI red teaming is structured adversarial testing of an AI system to find failures before an attacker does. For agents it means attempting to make the system take unauthorised actions through its tools, rather than only trying to make a model produce harmful text. Microsoft's AI red team distinguishes it from safety benchmarking, which measures harms you have already named, while red teaming exists to discover harms you have not.
How is agent red teaming different from model red teaming?
Model red teaming targets outputs. Agent red teaming targets actions taken with real credentials, so the unit of failure is a tool call rather than a sentence. Attacker entry points widen to any content the agent reads, including email, tickets, files and tool responses. Blast radius is bounded by the agent's permission set, which means identity and access control become part of the security test rather than a separate review.
How often should you red team an AI agent?
Run an automated injection and tool-abuse suite on every build in CI, review tools and permissions monthly, and run one manual exercise per quarter against your highest-permission agent. Add event triggers for any change that grants a new write capability, a new model version or a new data source. Calendar cadence alone fails because agent systems change their attack surface faster than a quarterly schedule tracks it.
Does the EU AI Act require red teaming?
Article 55 requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing. That obligation sits with model providers, not with every organisation deploying an agent built on someone else's model. If you are a deployer, the article is useful as a methodology reference and as a signal of where expectations are heading, but it is not the compliance trigger vendors sometimes present it as.
Do you need an external red team or can you do it in-house?
Do both, in that order. Buy the first quarterly exercise externally, because internal teams test the assumptions they already hold and the value of red teaming is the assumption you did not know you had made. Build the automated suite internally, since it runs on every commit and an external firm cannot own that loop. Use a different firm for any annual assurance work than the one running quarterly tests.
What should an AI red team report include?
Three numbers per agent, reported the same way each quarter. Maximum blast radius, named as a business outcome rather than a severity score. Hijack rate, meaning the share of scenarios where the team induced an unauthorised action, stated with the attempt count used. Median time to remediate, split by whether fixes were permission changes or prompt changes. Findings detail belongs in an appendix, not in the summary.
Where to start this week
Pick your highest-permission agent and write down, in one sentence, the worst action it can take without a human approving it. Then check whether anyone has ever tested that specific action under attack. In most organisations the answer to the second question is no, and the first sentence is uncomfortable enough on its own.
After that, add an attempt count to whatever test plan you already have. If your scenarios currently run once, run them 25 times and compare the result. That single change costs nothing and typically moves your reported exposure by more than any new tool would.
Related on agent operations
Testing is one leg of running agents in production. The other two are what on-call looks like when an agent runs the process and how governance maturity is actually staged.
References
- NIST, Technical Blog: Strengthening AI Agent Hijacking Evaluations, 17 January 2025. Used for the 11% to 81% attack success figures, the 57% to 80% attempt-count figures, the AgentDojo environments and the three added attack categories.
- Microsoft Security Blog, 3 takeaways from red teaming 100 generative AI products, 13 January 2025. Used for the operation counts and the distinction between red teaming and safety benchmarking.
- OWASP Gen AI Security Project, GenAI Exploit Round-up Report Q1 2026, 14 April 2026. Used for the eight catalogued incidents, the single CVE, and the incident categories.
- SANS Institute, 2026 SANS AI Survey announcement, 13 July 2026. Used for the governance role and audit framework figures. Sample: 536 practitioners and 57 senior leaders.
- OWASP Gen AI Security Project, Agentic AI: Threats and Mitigations. Used for the threat taxonomy structure.
- Help Net Security, Prompt injection still drives most agentic AI security failures in production, 11 June 2026. Used as corroboration on prompt injection prevalence.
- EU Artificial Intelligence Act, Article 55, obligations for providers of general-purpose AI models with systemic risk. Used for the adversarial testing obligation and who it binds.
Weakest part of this source base: the headline 11% to 81% result comes from one benchmark, one model version and one evaluation team, published January 2025. It shows that attack quality dominates outcomes. It does not measure any production agent, including yours.
Related reading