AI Code Security Scanners Compared: Four Tools Together Miss 61% of Real Flaws

Four static analysis tools, run together against production Java code with known vulnerabilities, found 38.8% of them. That figure comes from a peer reviewed benchmark, not a vendor page. AI code security scanners are now being asked to police a codebase that agents commit to 3 to 4 times faster than humans did. The distance between those two numbers is the entire buying decision.

What the scanner benchmarks actually measure

Static application security testing tools detect a minority of real vulnerabilities. A peer reviewed study presented at EASE 2024 ran CodeQL, Snyk Code, Semgrep and FindSecBugs against production Java code with known flaws. Individual detection ranged from 11.2% to 26.5%. All four together reached 38.8%.

Those numbers are far below what any product page implies, and they are not a scandal. They are what static analysis on real code looks like when someone counts honestly.

The same researchers then wrote new rules for Semgrep and pushed its detection to 44.7%, better than the four-tool combination. That result is the most useful thing in the paper. Tuning beat tool selection.

Synthetic suites and production code give different answers

Most published scanner comparisons run against the OWASP Benchmark or the Juliet test suite. Both are synthetic. Both are also public, which means vendors can and do tune against them.

Production code behaves differently. Framework indirection, custom sanitisers, dependency injection and generated boilerplate all break the data flow paths a rule expects to see. I would not buy a scanner on an OWASP Benchmark score, and I would treat a vendor citing one as telling me which test they trained for.

The least detected classes are the ones agents produce most

The EASE authors named the weak spots. Deserialization of untrusted data, input validation and cross-site scripting were among the least detected classes across all four tools.

Hold that list next to what the models generate. Veracode found cross-site scripting had a 15% security pass rate in generated code. The class scanners are worst at is the class models are worst at. That overlap is the actual finding of this post.

Why agent written code changed the denominator

Detection percentages were survivable when a team shipped 200 commits a week and a human read each one. Apiiro deployed its analysis engine across tens of thousands of repositories at Fortune 50 enterprises between December 2024 and June 2025. AI-assisted developers committed 3 to 4 times more code than their unassisted peers.

Monthly security findings at those organisations rose from roughly 1,000 to more than 10,000 over the same six months. Apiiro product manager Itay Nussbaum put it plainly to The Register: AI is multiplying not one kind of vulnerability, but all of them at once.

Review capacity did not move. That mismatch is the same one described in the analysis of how AI turned code review into the binding constraint, and a scanner does not fix it by finding more things.

The mix of flaws changed, not just the count

The composition shift is worse news than the volume. Syntax errors in AI-assisted code fell 76%. Logic bugs fell 60%. Those are the classes a compiler and a unit test already caught.

Meanwhile privilege escalation paths rose 322% and architectural design flaws rose 153%. Exposed cloud credentials appeared nearly twice as often. Those are the classes that need cross-file reasoning, which is exactly where pattern matching is weakest.

What the models actually ship

Veracode's Spring 2026 GenAI Code Security update is the best public measurement of the input side. It runs 80 coding tasks across Java, JavaScript, C# and Python, covering four CWE classes, with five task instances for each combination. More than 150 models have been evaluated.

The headline is that 55% of generation tasks produce secure code. The other 45% introduce a known flaw. Syntactic correctness now sits above 95%, and the security rate has barely moved in two years.

I do not think the model leaderboard matters much here. Syntactic quality and security quality are on different curves, and only one of them is improving.

Language choice moves the number more than model choice

Python scored a 62% security pass rate. C# reached 58% and JavaScript 57%. Java came in at 29%.

Model choice was worth less. Reasoning variants of GPT-5 reached 70% to 72%, the highest category Veracode tested, while other flagship releases clustered between 55% and 60%. Switching model buys perhaps 15 points. Switching from Java to Python is worth 33 in this data set, which is not a recommendation to rewrite anything, only a warning about which stack needs the tighter gate.

Two flaw classes carry almost all the failure

The aggregate hides the shape. SQL injection had an 82% security pass rate and insecure cryptographic algorithms 86%. Cross-site scripting scored 15%. Log injection scored 13%.

Models have absorbed the two flaw classes that every tutorial and every scanner has covered for 20 years. They have absorbed almost nothing about the two that get less attention. Your scanner configuration should reflect that asymmetry rather than the OWASP Top 10 in order.

Snyk, Semgrep and CodeQL on the axes that differ

Search demand for "Snyk vs Semgrep" assumes the two are substitutes competing on detection. They are not, and the comparison people should be running is different.

Prices are list, taken from each vendor's published pricing page in August 2026. Enterprise pricing is quoted, not published, and the discount at volume is real. Feature descriptions are from vendor documentation and have not been independently tested here.

What each tool is actually best at

Semgrep is the one you pick if you intend to write rules. The EASE result argues that is where the detection gain lives, and a team that never writes a custom rule is buying a fraction of the product.

Snyk is the one you pick if dependency risk is your bigger exposure. Given that roughly 20% of AI generated samples reference packages that do not exist, per the Cloud Security Alliance research note, that is not a small consideration. CodeQL is the one you pick if you are already inside GitHub and want the deepest data flow analysis without buying a second vendor.

List prices, and the unit that bites

Semgrep publishes Teams from $30 per contributor per month, covering one product, with secrets scanning at $15. Snyk publishes Team from $25 per contributing developer per month, and an Ignite tier from $1,260 per contributing developer per year.

The unit is the thing to check, not the rate. Both vendors bill on contributors who touched code in a period. Agents commit under human identities, so the count does not rise when an agent writes the code, but the volume it has to scan does. That is a better deal for you than a per-scan meter, and it is worth confirming in writing before the count is redefined at renewal.

Placement beats detection rate

On 23 June 2026, Snyk announced Evo Agentic Development Security and Semgrep announced Guardian. Same day, same idea. When two competitors ship the same architectural change in the same week, that is the market telling you where the problem moved.

The right comparison is not Snyk against Semgrep. It is inline against after the fact.

Inline enforcement inside the agent loop

Guardian bundles the Semgrep MCP server, hooks and skills into one install and scans every file an agent generates. When a finding appears, the agent is prompted to regenerate until the scan comes back clean or a human dismisses it. Semgrep reports running 3 million scans a week with 95% completing in under 5 seconds.

Snyk's version routes the same checks through an MCP server so the agent sees findings inside its own loop, and adds Agent Scan for the MCP servers and skills the agent itself depends on. That second piece matters more than it sounds, for reasons set out in the analysis of MCP servers as a supply chain risk. Snyk reports that 43% of developers run two or more AI coding environments at once.

The pull request gate is now one queue behind

GitHub published real numbers on the older model. During its public beta from May to July 2024, the median time to commit an Autofix suggestion for a pull request alert was 28 minutes, against 1.5 hours to fix the same alert manually. For SQL injection the gap was 18 minutes against 3.7 hours.

Now do the arithmetic those two sources imply. At 28 minutes per alert, the 10,000 monthly findings Apiiro measured would take about 4,700 engineer hours to clear. That is roughly 29 full-time engineers doing nothing else. Autofix made the per-alert cost three times better and the volume got ten times worse.

The false positive tax at agent volume

Every scanner vendor now sells an AI triage layer, because the backlog is the complaint. Semgrep says its layer, renamed Multimodal in March 2026, is over 95% accurate at categorising findings as false positives, and that initial autotriage cuts the backlog by roughly 60%. Snyk offers automatic fix generation with retesting through its own engine.

Take the 95% figure at face value and it is still a tax. Five percent of 10,000 monthly findings is 500 misclassifications, and the expensive direction of that error is a real flaw marked as noise. Neither vendor publishes the split between the two error types, which is the number that would actually settle this.

Here is my position. I would rather run a noisier scanner inline than a quieter one after the fact. An inline false positive costs the agent one regeneration, which is cheap and happens before anyone has context-switched. A missed finding at pull request time costs a production incident. Those are not comparable, and tuning for a low false positive rate optimises the wrong side of that trade.

Where this argument is weakest

Three problems with what I have just written, in order of how much they should bother you.

First, the detection figures are from 2024, cover Java only, and predate every product discussed in the placement section. Semgrep's cross-file Pro engine and Snyk Code's current engine were not tested in the configurations sold today. I am applying an old measurement to a new market because nothing better is public, and that is a genuine stretch rather than a rhetorical one.

Second, no independent party has tested Guardian against Evo on seeded vulnerabilities. Everything in the placement comparison comes from vendor announcements and vendor documentation. By this publication's own sourcing rules that is Tier 3 material, and it should not be the sole basis for a purchase.

Third, the strongest counter-argument is about adoption, not architecture. A pull request gate your team actually enforces beats an inline hook they disable in week two because it interrupts them. If your engineers have already turned off one linter this year, buy the gate. The Stanford study on whether users write more insecure code with AI assistants found participants who trusted the assistant less produced fewer vulnerabilities, which points at culture doing work no tool does. That gap between confidence and outcome is examined further in the piece on the distance between developer trust and developer usage.

One caveat on that study, stated where the finding is: it used 47 participants and a 2022 model, so treat the direction as informative and the magnitude as obsolete.

How to run the comparison yourself

No public benchmark will answer this for your codebase, and the vendors will not run the test you want. It takes one engineer about a week.

Record the seeds and the results in a file you keep. That is a small piece of first-party evidence, and it will outlast three vendor comparison pages. Publish the method if you can, since almost nobody has.

One design note. Have an agent write the seeded code rather than writing it yourself, using the same models and prompts your team uses. The point is to test detection against the flaw shapes your agents actually produce, not the shapes a security engineer thinks of, and those two sets overlap less than people expect. The wider question of whether agents behave differently in a real repository than in a benchmark is covered in the gap between coding agent scores and real repositories.

Frequently asked questions

Where to start this week

Start with a measurement rather than a purchase. Pull the count of open security findings in your repositories today and the count from six months ago. If the ratio looks anything like Apiiro's, your problem is throughput and no change of vendor will fix it.

Then move your existing scanner one position earlier. If you only have budget for one change this quarter, wiring the tool you already pay for into the agent loop will do more than replacing it with a tool that scores better on a synthetic benchmark. Both Snyk and Semgrep ship that integration for free with the licence you already hold.

Third, seed 20 vulnerabilities and run the test in the table above before your renewal date, so the conversation with the vendor starts from your numbers instead of theirs.

Related reading