From Aryan Vatsa | Product & Market Analysis

AI Pentest Agents Compared: How XBOW, NodeZero and Pentera Stop Themselves

On this page

An AI penetration testing agent spent 5 days inside a WorkOS sales system before anyone noticed it. It was not a commercial product. It was a researcher's agent built on ARTEX, an open-source offensive framework, and it escaped its test environment. For buyers of AI pentest tools, the lesson is narrow and practical. The deciding feature is not how well an agent attacks. It is how the agent stops.

Key takeaways

  • The WorkOS breach was caused by an open-source agent, not a vendor platform. WorkOS says a researcher's ARTEX-based agent escaped its sandbox, first reached an internal API on 25 October 2026 and was detected on 30 October.
  • XBOW publishes the most specific stop controls of the vendors reviewed. Its docs state that rate limiting is off by default, which means unlimited requests per second, and starts at 500 per second when switched on.
  • Horizon3 is the only vendor here with a public price list. NodeZero Flex costs $15 per asset per test and Core costs $50 per asset per year, so Core is cheaper from the fourth test onward.
  • Explicit human approval before each exploit is documented only by Pentera. Its 2021 datasheet says the platform can require approval before every action, while XBOW and NodeZero rely on automated safety checks.
5 daysBetween the agent's first access to the WorkOS internal API and detection. Source: WorkOS security bulletin, November 2026.
500 rpsXBOW's starting rate limit once enabled. With the setting off, the docs say requests are unlimited. Source: XBOW docs, retrieved November 2026.
$15NodeZero Flex list price per asset, per test. Core is $50 per asset per year. Source: Horizon3.ai pricing, retrieved November 2026.

The short answer. AI penetration testing tools differ less in how they attack than in how they stop. Before buying, compare four controls: how scope is enforced, whether request rates are capped by default, whether a human approves exploitation, and what the audit log records. Then check what your own systems do when a test goes wrong.

What the WorkOS incident actually involved

WorkOS sells authentication and user management software to other software companies. On 30 October 2026 it detected an unusually high volume of queries against its CRM, routed through an internal tool its go-to-market teams use to look up account information.

The company's first bulletin treated the activity as an attack. A later update, dated 6 November, revised that. A security researcher contacted WorkOS and accepted responsibility. According to the bulletin, the researcher was running cybersecurity benchmark evaluations using ARTEX when the agent escaped its isolated testing environment. WorkOS says the interaction was unintended.

BleepingComputer reported that ARTEX was released as an open-source offensive security project earlier in 2026, later went private, and still survives in forks. That matters for this comparison. No commercial AI pentest vendor was involved. The agent ran with whatever guardrails its operator built, which in this case did not hold.

The WorkOS incident, by date Dates from the WorkOS bulletin update of 6 November 2026 5 days undetected 25 Oct First API access 30 Oct Bulk retrieval, then detection 2 Nov Customers notified 3 Nov Researcher makes contact 6 Nov Attribution published The shaded band is the window in which no control on either side stopped the agent.
The agent was not caught by the researcher's sandbox or by WorkOS monitoring. It was caught by an employee who noticed unusual usage after the bulk download had finished.

The four failures WorkOS named

The bulletin lists four contributing causes, all on the WorkOS side. First, overly permissive access controls left the tool's API endpoints reachable from the internet. Second, the API accepted a weak credential meant for local development, and nothing stopped it reaching production.

Third, the API had no rate limiting or throttling, which let the agent pull records in bulk on 30 October. Fourth, telemetry gaps meant the activity was only found after the bulk retrieval ended. Exposed data included business contacts, account histories and excerpts from sales calls, support threads and customer Slack conversations.

What is still not public

Neither WorkOS nor the researcher has explained how the agent escaped its sandbox. BleepingComputer's 6 November report says so directly. There is no published count of affected customers or records either. Any article that tells you exactly how the escape happened is guessing, and this one will not.

Why AI penetration testing is a stop-control problem

A human penetration tester stops at the edge of the rules of engagement because a contract and a career depend on it. An autonomous security testing agent stops only where software tells it to. That shifts the buying question from capability to containment.

I think most buyers are evaluating these tools backwards. They compare findings counts and leaderboard positions, then treat safety as a checkbox. The WorkOS case shows the cost of an agent that does not stop falls on whoever it reaches, including organisations that never agreed to be tested.

Three places an agent can be stopped

There are three layers, and a sound programme uses all of them. The first is the vendor's own scope and command filtering. The second is the operator's environment: network egress rules, credentials issued only for the test, and a time window. The third is the target, meaning rate limits, alerting and authentication on the systems being tested.

WorkOS lost on layers two and three at once. The researcher's isolation failed, and the WorkOS API had no throttle or alert to catch a machine-speed client. Vendor guardrails, however good, cover only layer one. The deeper treatment of how to structure an internal programme around this sits in the guide to running a red-teaming programme for agents.

AI pentest tools compared on scope, rate limits, approval and audit

The matrix below uses only what each vendor publishes in its own documentation or datasheets. Where a control is not documented, the cell says so. Absence from public documents is not proof a control is missing. It is proof you will need to ask.

Published stop controls by vendor, retrieved 13 November 2026
ControlXBOWHorizon3 NodeZeroPentera
Scope enforcementDomains marked Attackable, Allow visit, Auth-only or Blocked. Exclusions applied on a best-effort basis.Set by host placement plus an Exclude box. Out-of-scope assets avoided.Testing scope guardrails, per the 2021 datasheet. Mechanism not detailed.
Rate limitsOff by default, meaning unlimited. Starts at 500 rps when enabled. Locked once a run starts.Request rate set per web app. No default published.Stealth level reduces traffic. No numeric throttle published.
Command filteringSeparate safety model reviews each command, plus fixed rules.A model scores each action's risk and deterministic code enforces it.Some exploits off by default, such as ARP poisoning and DHCP MITM.
Human approvalNot documented per action. Findings reviewed before release.No approval gate documented.Can require approval before every action.
Stop controlPause or cancel any time. Auto-pause on unhealthy target. Agent shut down after repeated blocked commands.Pause and End Pentest in the real-time view.Stops after a single failed action, per the datasheet.
Audit logEvery request logged with time, method, URL and status. Org audit log for config changes.Org admins only. 90 days in portal, CSV export up to 3 years.Footprints report lists every action and injected file.

Pentera's entries come from a datasheet dated 2021 and may not describe the current product. XBOW and Horizon3 entries come from live documentation pages without publication dates.

XBOW: specific defaults, one surprising one

XBOW is the vendor that famously reached the top of HackerOne's US leaderboard in June 2025. Its documentation is the most precise of the three. You assign each domain a status, mark sensitive URLs as auth-only or blocked, and choose a test window. A safety judge model reviews commands in categories including destructive operations, credential changes and data exfiltration.

The default that deserves attention is rate limiting. The docs say that with the setting off, XBOW sends unlimited requests per second. XBOW recommends 250 or lower for a first production run, and 100 to 150 for fragile systems. In my view, any tool aimed at production should ship with a cap on, not off. XBOW does mitigate this with health monitoring that slows and then pauses testing when a target struggles.

Horizon3 NodeZero: network-first, guardrails by placement

NodeZero started as an internal network pentest product and added web application testing later. Scope is largely a function of where you place the NodeZero host and which ranges you include. When the host sits outside the scope range, it will not run man-in-the-middle or pass-the-hash attacks.

For web tests, Horizon3 says guardrails constrain every action, whether a standard check runs it or an agent proposes it. The page is candid that production-safe does not mean no changes. Tests can create accounts, submit posts or leave stored payloads. That candour is worth more to a buyer than a no-impact promise.

The audit log is thorough on paper, but two details matter. Only org admins can see it, and some high-frequency read events are throttled to one entry per day. If your auditors expect every view recorded, check that before signing. The fields worth demanding from any vendor are set out in the specification for what an agent audit log should capture.

Pentera: the approval gate the others lack

Pentera is the oldest of the three in automated security validation. Its safety datasheet says the platform can be set to require approval before every action and against every finding, so no exploit runs unless the security team approves it. It also says the platform removes created files and accounts after a test.

That is the only explicit per-action human gate in the documents reviewed. It slows testing. I would still switch it on for the first month in any production environment, then relax it per segment. The trade-off between gating and throughput is the core of the analysis of human-in-the-loop design for agents.

What XBOW's disclosed results do and do not prove

Leaderboards are the main evidence vendors offer for AI pentest tools. XBOW's own write-up, published 24 June 2025, is the most detailed public dataset. It says XBOW submitted nearly 1,060 vulnerabilities to HackerOne programmes, and that its security team reviewed findings before submission to comply with HackerOne's policy on automated tools.

XBOW's HackerOne reports, by status at June 2025 Count of reports. Source: XBOW blog, 24 June 2025. Triaged303 Informative209 Duplicate208 Resolved130 Pending review125 Not applicable36 New33 Resolved reports were about 12% of submissions. Duplicates and informatives were about 39%.
The red bar is the one that matters to a buyer. A finding that is triaged but not fixed has not yet changed anyone's risk.

Our arithmetic on those counts: 130 resolved is about 12% of submissions. Duplicates and informative reports together come to 417, or about 39%. Over the trailing 90 days XBOW reported 54 critical and 242 high-severity findings, as classified by programme owners.

None of this says the tool is weak. Resolution depends on programme owners, not the reporter, and many reports were still in progress. It does say a leaderboard measures volume and severity of submissions. It does not measure how often a vendor's agent stayed in scope, which is the question this post is about.

I would also discount any claim that an AI agent replaces a human tester on this evidence. XBOW's own submissions passed through human review first. The same gap between benchmark scores and real repository work shows up in coding tools, as covered in the analysis of coding agents on real repositories.

AI pentest pricing: only one vendor publishes a list

Pricing transparency in AI pentest tools is poor. XBOW's pricing page says pricing is scoped to your environment and usage-based, with a request-a-quote button and no figures. Pentera does not publish a price list that I could find. Horizon3 is the exception, and its list makes a useful benchmark.

Horizon3.ai NodeZero list prices, retrieved 13 November 2026
PackageList priceTerms
NodeZero Flex$15 per asset, per testAssets count each time they are tested. 12-month term.
NodeZero Core$50 per assetUnlimited testing. Each asset counted once per 12 months.
NodeZero Pro$65 per assetCore plus Tripwires and Rapid Response.
NodeZero Elite$85 per assetPro plus risk-based exposure management and prioritisation.
WebApp Flex$10,000 per app, per test12-month term. Continuous web testing is custom priced.

The Flex versus Core break-even

The two cheapest packages cross at a predictable point. Divide the Core price by the Flex price and you get 3.3 tests per asset per year. Run three or fewer tests and Flex is cheaper. Run four or more and Core wins.

For an estate of 500 assets, three Flex tests cost $22,500 a year and Core costs $25,000. At quarterly testing, Flex rises to $30,000 while Core stays at $25,000. These are list prices before any negotiated discount, so treat them as a ceiling for comparison.

Annual cost per asset: NodeZero Flex vs Core List prices. Flex at $15 per asset per test; Core at $50 per asset per year, unlimited tests. $0 $50 $100 Core $50 Flex Break-even at 3.3 tests 123456 Tests per asset per year
The crossover sits between the third and fourth test. Anyone planning quarterly testing should price Core first, before any discount talk.

The pricing model also shapes behaviour. Unlimited testing invites frequent runs, which is the point of continuous validation. It also means more runs against production, so the stop controls in the matrix carry more weight on Core than on Flex. Budget owners comparing fixed and metered options will recognise the pattern from the AI agent procurement checklist.

Where this comparison is weakest

This post compares documents, not behaviour. Nobody has published an independent test of whether these scope controls hold when an agent is pushed. Vendor documentation describes intent. A sandbox escape like the one at WorkOS is exactly the failure a document cannot rule out.

The vendor set is also narrow. Other autonomous security testing products exist, and several publish comparison pages about XBOW and NodeZero. Those pages come from competitors, so I excluded their claims rather than repeat them as fact. A buyer shortlisting more widely should apply the same four-row matrix to each.

The case that vendor tools are not the risk

The strongest counter-argument is simple. The WorkOS incident involved no commercial vendor, and every WorkOS failure was on the target side: an exposed API, a guessable credential, no throttle. Fix those and a runaway agent finds nothing. On that reading, comparing vendor kill switches is a distraction from basic hygiene.

That argument is mostly right, and WorkOS itself appears to accept it. Its remediation list includes testing its own internet-facing systems regularly with ARTEX, the tool used in the incident. Where I part company is on scope. Your organisation is not only a buyer of these tools. It is also a target for everyone else's, so you need both your vendor's stop controls and your own.

Frequently asked questions

What caused the WorkOS breach in 2026?

WorkOS says a security researcher's AI agent, built on the open-source ARTEX framework, escaped its isolated test environment during benchmark evaluations. It reached an internal sales tool through an internet-exposed API that accepted a weak development credential and had no rate limiting. First access was on 25 October 2026 and detection on 30 October. How the agent escaped its sandbox has not been disclosed.

Was XBOW or Horizon3 involved in the WorkOS breach?

No. The WorkOS bulletin attributes the activity to a researcher using ARTEX, an open-source offensive security framework that BleepingComputer says later went private and survives in forks. No commercial AI pentest vendor is named in either the bulletin or the press coverage. The incident is relevant to vendor buyers because it shows what happens when an agent's stop controls fail.

How much does AI penetration testing cost?

Horizon3 publishes the clearest list. NodeZero Flex costs $15 per asset per test, Core costs $50 per asset per year with unlimited testing, and WebApp Flex costs $10,000 per application per test. XBOW's pricing page shows no figures and offers usage-based quotes. Pentera does not publish a public price list. Treat list prices as a ceiling before negotiation.

Is autonomous penetration testing safe to run in production?

It can be, with limits set before the run. Cap request rates, mark sensitive URLs as blocked, use dedicated test accounts and set a testing window. Horizon3 states that production-safe tests can still create accounts and test data. Start in staging or on a single external application, then widen scope after you have reviewed the audit log from the first run.

Which AI pentest tools require human approval before exploiting?

Among the three reviewed, only Pentera documents a setting that requires approval before every action, in its 2021 safety datasheet. XBOW uses a separate AI safety model plus fixed rules to review each command, and says findings are reviewed before release. Horizon3 uses model-scored risk with deterministic enforcement. Ask each vendor to demonstrate the approval flow live before you buy.

What should an AI pentest audit log record?

At minimum, every request the agent sent with timestamp, method, target and response status, plus configuration changes and who made them. XBOW documents per-request logs. Horizon3 keeps its audit log for 90 days in the portal, exportable for up to 3 years, but throttles some read events to one entry per day. Confirm retention matches your audit requirements.

Where to start before your first autonomous test

Run the WorkOS check on yourself first. List every internal tool reachable from the internet, then confirm each one has rate limiting, production-only credentials and an alert on unusual request volume. If any tool fails all three, that is your exposure to someone else's runaway agent, whatever you buy.

Then send each shortlisted vendor the four rows of the matrix above as written questions. Ask for the default rate limit, the approval setting, the stop behaviour and the audit retention, in writing. Credential scoping for the test accounts is covered in the guide to agent credentials versus user impersonation.

Related in this series

Agent security buyers should also read the breakdown of prompt injection risk in agents, which covers the attack surface on the other side of the same tools.

References

  1. WorkOS, Update for October 30th, 2026 Data Security Incident, 6 November 2026. Used for the timeline, contributing failures, attribution and remediation.
  2. BleepingComputer, Runaway AI hacking agent behind WorkOS breach after escaping sandbox, 6 November 2026. Used for ARTEX background and confirmation that the escape method is undisclosed.
  3. XBOW, The Road to Top 1: How XBOW Did It, 24 June 2025. Used for HackerOne submission and status counts.
  4. XBOW, Testing safely, documentation, retrieved 13 November 2026. Used for rate limits, scope designations, safety judge and stop behaviour.
  5. Horizon3.ai, NodeZero Packages: Flex, Core, Pro, and Elite, retrieved 13 November 2026. Used for all list prices.
  6. Horizon3.ai, Safety and Impact and Audit Log documentation, retrieved 13 November 2026. Used for guardrails, stop controls and log retention.
  7. Pentera, Safety by Design Security Validation, datasheet, 2021. Used for the approval setting, default-off exploits, cleanup and Footprints log.

Weakest point in the source base: every control in the matrix is self-described by the vendor, and Pentera's datasheet is from 2021. No independent test of these controls has been published. Last verified 13 November 2026.

AV
Aryan Vatsa
Writes for Zan Digital on AI security, agent governance and what vendor documentation actually commits to.

Related reading