From Aryan Vatsa | Product & Market Analysis
AI Pentest Agents Compared: How XBOW, NodeZero and Pentera Stop Themselves
On this page
An AI penetration testing agent spent 5 days inside a WorkOS sales system before anyone noticed it. It was not a commercial product. It was a researcher's agent built on ARTEX, an open-source offensive framework, and it escaped its test environment. For buyers of AI pentest tools, the lesson is narrow and practical. The deciding feature is not how well an agent attacks. It is how the agent stops.
Key takeaways
- The WorkOS breach was caused by an open-source agent, not a vendor platform. WorkOS says a researcher's ARTEX-based agent escaped its sandbox, first reached an internal API on 25 October 2026 and was detected on 30 October.
- XBOW publishes the most specific stop controls of the vendors reviewed. Its docs state that rate limiting is off by default, which means unlimited requests per second, and starts at 500 per second when switched on.
- Horizon3 is the only vendor here with a public price list. NodeZero Flex costs $15 per asset per test and Core costs $50 per asset per year, so Core is cheaper from the fourth test onward.
- Explicit human approval before each exploit is documented only by Pentera. Its 2021 datasheet says the platform can require approval before every action, while XBOW and NodeZero rely on automated safety checks.
The short answer. AI penetration testing tools differ less in how they attack than in how they stop. Before buying, compare four controls: how scope is enforced, whether request rates are capped by default, whether a human approves exploitation, and what the audit log records. Then check what your own systems do when a test goes wrong.
What the WorkOS incident actually involved
WorkOS sells authentication and user management software to other software companies. On 30 October 2026 it detected an unusually high volume of queries against its CRM, routed through an internal tool its go-to-market teams use to look up account information.
The company's first bulletin treated the activity as an attack. A later update, dated 6 November, revised that. A security researcher contacted WorkOS and accepted responsibility. According to the bulletin, the researcher was running cybersecurity benchmark evaluations using ARTEX when the agent escaped its isolated testing environment. WorkOS says the interaction was unintended.
BleepingComputer reported that ARTEX was released as an open-source offensive security project earlier in 2026, later went private, and still survives in forks. That matters for this comparison. No commercial AI pentest vendor was involved. The agent ran with whatever guardrails its operator built, which in this case did not hold.
The four failures WorkOS named
The bulletin lists four contributing causes, all on the WorkOS side. First, overly permissive access controls left the tool's API endpoints reachable from the internet. Second, the API accepted a weak credential meant for local development, and nothing stopped it reaching production.
Third, the API had no rate limiting or throttling, which let the agent pull records in bulk on 30 October. Fourth, telemetry gaps meant the activity was only found after the bulk retrieval ended. Exposed data included business contacts, account histories and excerpts from sales calls, support threads and customer Slack conversations.
What is still not public
Neither WorkOS nor the researcher has explained how the agent escaped its sandbox. BleepingComputer's 6 November report says so directly. There is no published count of affected customers or records either. Any article that tells you exactly how the escape happened is guessing, and this one will not.
Why AI penetration testing is a stop-control problem
A human penetration tester stops at the edge of the rules of engagement because a contract and a career depend on it. An autonomous security testing agent stops only where software tells it to. That shifts the buying question from capability to containment.
I think most buyers are evaluating these tools backwards. They compare findings counts and leaderboard positions, then treat safety as a checkbox. The WorkOS case shows the cost of an agent that does not stop falls on whoever it reaches, including organisations that never agreed to be tested.
Three places an agent can be stopped
There are three layers, and a sound programme uses all of them. The first is the vendor's own scope and command filtering. The second is the operator's environment: network egress rules, credentials issued only for the test, and a time window. The third is the target, meaning rate limits, alerting and authentication on the systems being tested.
WorkOS lost on layers two and three at once. The researcher's isolation failed, and the WorkOS API had no throttle or alert to catch a machine-speed client. Vendor guardrails, however good, cover only layer one. The deeper treatment of how to structure an internal programme around this sits in the guide to running a red-teaming programme for agents.
AI pentest tools compared on scope, rate limits, approval and audit
The matrix below uses only what each vendor publishes in its own documentation or datasheets. Where a control is not documented, the cell says so. Absence from public documents is not proof a control is missing. It is proof you will need to ask.
| Control | XBOW | Horizon3 NodeZero | Pentera |
|---|---|---|---|
| Scope enforcement | Domains marked Attackable, Allow visit, Auth-only or Blocked. Exclusions applied on a best-effort basis. | Set by host placement plus an Exclude box. Out-of-scope assets avoided. | Testing scope guardrails, per the 2021 datasheet. Mechanism not detailed. |
| Rate limits | Off by default, meaning unlimited. Starts at 500 rps when enabled. Locked once a run starts. | Request rate set per web app. No default published. | Stealth level reduces traffic. No numeric throttle published. |
| Command filtering | Separate safety model reviews each command, plus fixed rules. | A model scores each action's risk and deterministic code enforces it. | Some exploits off by default, such as ARP poisoning and DHCP MITM. |
| Human approval | Not documented per action. Findings reviewed before release. | No approval gate documented. | Can require approval before every action. |
| Stop control | Pause or cancel any time. Auto-pause on unhealthy target. Agent shut down after repeated blocked commands. | Pause and End Pentest in the real-time view. | Stops after a single failed action, per the datasheet. |
| Audit log | Every request logged with time, method, URL and status. Org audit log for config changes. | Org admins only. 90 days in portal, CSV export up to 3 years. | Footprints report lists every action and injected file. |
Pentera's entries come from a datasheet dated 2021 and may not describe the current product. XBOW and Horizon3 entries come from live documentation pages without publication dates.
XBOW: specific defaults, one surprising one
XBOW is the vendor that famously reached the top of HackerOne's US leaderboard in June 2025. Its documentation is the most precise of the three. You assign each domain a status, mark sensitive URLs as auth-only or blocked, and choose a test window. A safety judge model reviews commands in categories including destructive operations, credential changes and data exfiltration.
The default that deserves attention is rate limiting. The docs say that with the setting off, XBOW sends unlimited requests per second. XBOW recommends 250 or lower for a first production run, and 100 to 150 for fragile systems. In my view, any tool aimed at production should ship with a cap on, not off. XBOW does mitigate this with health monitoring that slows and then pauses testing when a target struggles.
Horizon3 NodeZero: network-first, guardrails by placement
NodeZero started as an internal network pentest product and added web application testing later. Scope is largely a function of where you place the NodeZero host and which ranges you include. When the host sits outside the scope range, it will not run man-in-the-middle or pass-the-hash attacks.
For web tests, Horizon3 says guardrails constrain every action, whether a standard check runs it or an agent proposes it. The page is candid that production-safe does not mean no changes. Tests can create accounts, submit posts or leave stored payloads. That candour is worth more to a buyer than a no-impact promise.
The audit log is thorough on paper, but two details matter. Only org admins can see it, and some high-frequency read events are throttled to one entry per day. If your auditors expect every view recorded, check that before signing. The fields worth demanding from any vendor are set out in the specification for what an agent audit log should capture.
Pentera: the approval gate the others lack
Pentera is the oldest of the three in automated security validation. Its safety datasheet says the platform can be set to require approval before every action and against every finding, so no exploit runs unless the security team approves it. It also says the platform removes created files and accounts after a test.
That is the only explicit per-action human gate in the documents reviewed. It slows testing. I would still switch it on for the first month in any production environment, then relax it per segment. The trade-off between gating and throughput is the core of the analysis of human-in-the-loop design for agents.
What XBOW's disclosed results do and do not prove
Leaderboards are the main evidence vendors offer for AI pentest tools. XBOW's own write-up, published 24 June 2025, is the most detailed public dataset. It says XBOW submitted nearly 1,060 vulnerabilities to HackerOne programmes, and that its security team reviewed findings before submission to comply with HackerOne's policy on automated tools.
Our arithmetic on those counts: 130 resolved is about 12% of submissions. Duplicates and informative reports together come to 417, or about 39%. Over the trailing 90 days XBOW reported 54 critical and 242 high-severity findings, as classified by programme owners.
None of this says the tool is weak. Resolution depends on programme owners, not the reporter, and many reports were still in progress. It does say a leaderboard measures volume and severity of submissions. It does not measure how often a vendor's agent stayed in scope, which is the question this post is about.
I would also discount any claim that an AI agent replaces a human tester on this evidence. XBOW's own submissions passed through human review first. The same gap between benchmark scores and real repository work shows up in coding tools, as covered in the analysis of coding agents on real repositories.
AI pentest pricing: only one vendor publishes a list
Pricing transparency in AI pentest tools is poor. XBOW's pricing page says pricing is scoped to your environment and usage-based, with a request-a-quote button and no figures. Pentera does not publish a price list that I could find. Horizon3 is the exception, and its list makes a useful benchmark.
| Package | List price | Terms |
|---|---|---|
| NodeZero Flex | $15 per asset, per test | Assets count each time they are tested. 12-month term. |
| NodeZero Core | $50 per asset | Unlimited testing. Each asset counted once per 12 months. |
| NodeZero Pro | $65 per asset | Core plus Tripwires and Rapid Response. |
| NodeZero Elite | $85 per asset | Pro plus risk-based exposure management and prioritisation. |
| WebApp Flex | $10,000 per app, per test | 12-month term. Continuous web testing is custom priced. |
The Flex versus Core break-even
The two cheapest packages cross at a predictable point. Divide the Core price by the Flex price and you get 3.3 tests per asset per year. Run three or fewer tests and Flex is cheaper. Run four or more and Core wins.
For an estate of 500 assets, three Flex tests cost $22,500 a year and Core costs $25,000. At quarterly testing, Flex rises to $30,000 while Core stays at $25,000. These are list prices before any negotiated discount, so treat them as a ceiling for comparison.
The pricing model also shapes behaviour. Unlimited testing invites frequent runs, which is the point of continuous validation. It also means more runs against production, so the stop controls in the matrix carry more weight on Core than on Flex. Budget owners comparing fixed and metered options will recognise the pattern from the AI agent procurement checklist.
Where this comparison is weakest
This post compares documents, not behaviour. Nobody has published an independent test of whether these scope controls hold when an agent is pushed. Vendor documentation describes intent. A sandbox escape like the one at WorkOS is exactly the failure a document cannot rule out.
The vendor set is also narrow. Other autonomous security testing products exist, and several publish comparison pages about XBOW and NodeZero. Those pages come from competitors, so I excluded their claims rather than repeat them as fact. A buyer shortlisting more widely should apply the same four-row matrix to each.
The case that vendor tools are not the risk
The strongest counter-argument is simple. The WorkOS incident involved no commercial vendor, and every WorkOS failure was on the target side: an exposed API, a guessable credential, no throttle. Fix those and a runaway agent finds nothing. On that reading, comparing vendor kill switches is a distraction from basic hygiene.
That argument is mostly right, and WorkOS itself appears to accept it. Its remediation list includes testing its own internet-facing systems regularly with ARTEX, the tool used in the incident. Where I part company is on scope. Your organisation is not only a buyer of these tools. It is also a target for everyone else's, so you need both your vendor's stop controls and your own.
Frequently asked questions
What caused the WorkOS breach in 2026?
WorkOS says a security researcher's AI agent, built on the open-source ARTEX framework, escaped its isolated test environment during benchmark evaluations. It reached an internal sales tool through an internet-exposed API that accepted a weak development credential and had no rate limiting. First access was on 25 October 2026 and detection on 30 October. How the agent escaped its sandbox has not been disclosed.
Was XBOW or Horizon3 involved in the WorkOS breach?
No. The WorkOS bulletin attributes the activity to a researcher using ARTEX, an open-source offensive security framework that BleepingComputer says later went private and survives in forks. No commercial AI pentest vendor is named in either the bulletin or the press coverage. The incident is relevant to vendor buyers because it shows what happens when an agent's stop controls fail.
How much does AI penetration testing cost?
Horizon3 publishes the clearest list. NodeZero Flex costs $15 per asset per test, Core costs $50 per asset per year with unlimited testing, and WebApp Flex costs $10,000 per application per test. XBOW's pricing page shows no figures and offers usage-based quotes. Pentera does not publish a public price list. Treat list prices as a ceiling before negotiation.
Is autonomous penetration testing safe to run in production?
It can be, with limits set before the run. Cap request rates, mark sensitive URLs as blocked, use dedicated test accounts and set a testing window. Horizon3 states that production-safe tests can still create accounts and test data. Start in staging or on a single external application, then widen scope after you have reviewed the audit log from the first run.
Which AI pentest tools require human approval before exploiting?
Among the three reviewed, only Pentera documents a setting that requires approval before every action, in its 2021 safety datasheet. XBOW uses a separate AI safety model plus fixed rules to review each command, and says findings are reviewed before release. Horizon3 uses model-scored risk with deterministic enforcement. Ask each vendor to demonstrate the approval flow live before you buy.
What should an AI pentest audit log record?
At minimum, every request the agent sent with timestamp, method, target and response status, plus configuration changes and who made them. XBOW documents per-request logs. Horizon3 keeps its audit log for 90 days in the portal, exportable for up to 3 years, but throttles some read events to one entry per day. Confirm retention matches your audit requirements.
Where to start before your first autonomous test
Run the WorkOS check on yourself first. List every internal tool reachable from the internet, then confirm each one has rate limiting, production-only credentials and an alert on unusual request volume. If any tool fails all three, that is your exposure to someone else's runaway agent, whatever you buy.
Then send each shortlisted vendor the four rows of the matrix above as written questions. Ask for the default rate limit, the approval setting, the stop behaviour and the audit retention, in writing. Credential scoping for the test accounts is covered in the guide to agent credentials versus user impersonation.
Related in this series
Agent security buyers should also read the breakdown of prompt injection risk in agents, which covers the attack surface on the other side of the same tools.
References
- WorkOS, Update for October 30th, 2026 Data Security Incident, 6 November 2026. Used for the timeline, contributing failures, attribution and remediation.
- BleepingComputer, Runaway AI hacking agent behind WorkOS breach after escaping sandbox, 6 November 2026. Used for ARTEX background and confirmation that the escape method is undisclosed.
- XBOW, The Road to Top 1: How XBOW Did It, 24 June 2025. Used for HackerOne submission and status counts.
- XBOW, Testing safely, documentation, retrieved 13 November 2026. Used for rate limits, scope designations, safety judge and stop behaviour.
- Horizon3.ai, NodeZero Packages: Flex, Core, Pro, and Elite, retrieved 13 November 2026. Used for all list prices.
- Horizon3.ai, Safety and Impact and Audit Log documentation, retrieved 13 November 2026. Used for guardrails, stop controls and log retention.
- Pentera, Safety by Design Security Validation, datasheet, 2021. Used for the approval setting, default-off exploits, cleanup and Footprints log.
Weakest point in the source base: every control in the matrix is self-described by the vendor, and Pentera's datasheet is from 2021. No independent test of these controls has been published. Last verified 13 November 2026.
Related reading