From Ritu Raj | Product & Market Analysis
AI Phishing Click Rates: 8 Tests Put AI Near or Below Humans, Not 4.5x Above
On this page
AI phishing is widely reported as 4.5 times more effective than ordinary phishing, on a 54% versus 12% click rate. The largest test that pits AI against human writers on the same recipients found 12.15% versus 10.92%, across 9,129 people. Eight head-to-head tests point the same way: AI writes about as well as people, and sometimes worse. What changed is the price.
Key takeaways
- No head-to-head test reproduces a 4.5x advantage over human writers. Across 8 tests, the pooled gap runs from 39 points behind human writers to 1.2 points ahead.
- The 54% figure matches a study of 101 people, in which human experts also scored 54%. Microsoft's numbers match that paper on all three headline figures, though we could not confirm the citation in the report itself.
- The lure moves clicks more than the author does. In a 9,129-person university test, the choice of lure changed click rates about 4.5 times over, while authorship changed the pooled rate by 1.23 points.
- Cost is the measured shift, not persuasion. A 7,741-person USENIX experiment put automated personalisation at about $0.03 per email, and it nearly tripled clicks over generic phishing.
The AI phishing claim that travels furthest
Is AI phishing more effective than human phishing? On the controlled evidence, no. Head-to-head tests show AI-written email roughly level with human writing or behind it. The strong result is against generic spam. AI lowers the cost of personalisation, and personalisation is what raises clicks, so the threat is real and the 4.5x framing is not.
The version most buyers meet is shorter. The Register's report on Microsoft's Digital Defense Report 2025 says AI-automated emails drew 54 percent click-through against 12 percent for "non-AI phishes". A Microsoft news release calls that 4.5 times higher than traditional methods.
Each retelling hardens the comparison. A June 2026 Cloud Security Alliance research note says "testing across enterprise environments" found 54% for AI against 12% for "manually crafted equivalents". That sentence describes an enterprise trial of AI against human writers. I could not find such a trial, and the paper that matches those numbers is neither of those things.
This matters to your budget. A security leader who believes AI multiplies click rates 4.5 times will buy differently from one who believes AI makes competent phishing cheap. The first buys detection for better prose. The second plans for more volume and more personalisation. The gap between INTERPOL's 55% and the FBI's 2% shows the same pattern in crime statistics: the definition does most of the work.
The AI phishing study table: 8 tests side by side
Eight tests below compare AI-written phishing against a human-written or generic alternative, with real clicks on real links. Five come from academic teams and 3 from companies that sell security services. Read the sample column first, because it decides how much weight each row can carry.
| Source | Method | Sample | Click rate | Main caveat |
|---|---|---|---|---|
| Heiding et al., 2024 preprint, later in Expert Systems with Applications | Randomised, 4 arms, Claude 3.5 Sonnet | 101 university affiliates, about 25 per arm | Control 12%, human expert 54%, AI 54%, hybrid 56% | Volunteers paid a $5 gift card. Standard errors near 12 points. |
| Heiding et al., 2023 preprint | Randomised, 4 arms, GPT-4 | 112 students, about 28 per arm | Control 19-28%, GPT 30-44%, human V-Triad 69-79%, combined 43-81% | GPT email did not name the university, so arms differ in personalisation too. |
| Bethany et al., 2025, IEEE Access | Field test, 3 lures, GPT-4 | 9,129 staff and student workers at one university | Human 10.92%, LLM 12.15% | One institution. No informed consent. |
| Czybik et al., 2026, USENIX Security | Field test, generic vs personalised, human vs LLM | 7,741 participants | Generic 3.7-3.9%, LLM personalised 10.0%, human personalised 24.2% | Forewarned, trained participants. Simple local models. |
| Weinz et al., 2025, ACM ASIA CCS | Simulations at 3 firms, GPT-3.5 Turbo | 71,309 emails sent | Landing page visits per email sent: button 5.2%, LLM 3.4% (our calculation) | Paper's metric is visits per email opened. Mainly Central Europe. |
| IBM X-Force, 2023 | Vendor test, 1 email per arm, ChatGPT | 1,600 staff at one healthcare firm | Human 14%, AI 11% | Single email each, so the result may be about that email. |
| SoSafe, 2023 | Vendor simulations, ChatGPT-3.5 | About 1,500 simulated attacks, per trade press | Human 27%, AI 21% | Non-personalised. The release states no sample. |
| Hoxhunt, March 2025 | Vendor platform, AI agent vs red team | 70,000 AI simulations, drawn from 2.5M users | Human 2.25%, AI 2.78% | Human arm size unpublished. All users trained on the vendor's product. |
Click rates are not comparable between rows. Populations, lures and the definition of a click all differ. Compare inside a row, never across rows. The range in the Heiding 2023 row is the paper's own and appears to reflect all participants against active participants only. The Czybik generic rate differs between the paper excerpts we could read, as explained below.
Where the 54% click rate seems to start
The 54% figure appears to trace to a 101-person study, not to enterprise telemetry. Heiding, Lermen, Kao, Schneier and Vishwanath posted their preprint on 30 November 2024. It reports a control arm at 12%, human experts at 54%, fully automated AI at 54%, and a hybrid at 56%. It also says AI could raise profitability by up to 50 times.
Microsoft's reported figures are 54%, 12% and up to 50-fold. Those are three matches, and the preprint falls inside the report's July 2024 to June 2025 window. A Microsoft release says AI phishing could lift the profitability of attacks "by up to 50-fold", and the same ratio follows directly: 54 divided by 12 is 4.5.
State this carefully, because I would hold anyone else to the same bar. We could not extract the footnote from Microsoft's report, because the PDF would not parse. Neither the release nor the press coverage we opened names a source. So this is a strong match, not a confirmed citation. Microsoft may hold separate data that lands on identical numbers, though I think that is unlikely.
If the match holds, the comparison flips
The paper's 12% is not human phishing. The authors built it from spam in their own inboxes, edited to be less suspicious. The paper's human comparison is the 54% arm, where AI tied. So the paper's actual finding is close to the opposite of "4.5 times better than people". It says automation reached expert level, and that both beat an arbitrary template by a wide margin.
What 25 people per arm can and cannot show
The preprint's results table gives the AI arm a standard error of 11.8 points and the human arm 12.2. Those are large. Working from them, a normal approximation puts the AI arm's 95% interval at about 31% to 77%. The AI minus human gap is -0.4 points, with a combined standard error near 17, which gives an interval of about 33 points in either direction.
So the data cannot separate "AI slightly ahead" from "AI far behind". It does separate AI from the control. Nobody should quote 54% as a population rate, since participants were university affiliates who volunteered through flyers and mailing lists and accepted a gift card. The authors' figure tells you a personalised email can work on a small, willing group.
What the large AI phishing field experiments measured
Three academic field tests have samples in the thousands, and none shows a large AI writing advantage overall.
Bethany: a small lead on 9,129 recipients
Bethany and colleagues at the University of Texas at San Antonio sent lateral phishing to a university workforce, spoofing internal addresses. Human-written mail drew 508 clicks from 4,652 recipients. GPT-4 mail drew 544 from 4,477. That is 10.92% against 12.15%.
Our own two-proportion check puts that 1.23-point gap at a 95% interval of about -0.1 to +2.5 points, so it sits right at the edge of the usual 5% threshold. The paper reports significance for one lure, not all three. Treat it as a small LLM lead at most, and as parity with slight upside for the attacker.
Czybik: personalisation is the multiplier
The largest study is Czybik and colleagues at USENIX Security 2026, with 7,741 participants. Automated web searches built a profile of each target, and an LLM wrote the email. The abstract says this "almost triples" clicks over generic phishing, whether the generic email was human or LLM written.
Excerpts of the paper put LLM personalised email at 10.0% and manually crafted spear phishing at 24.2%. So a skilled human still wins by 14.2 points. Be careful with the headline ratio. The institute's release says AI personalisation reached 2.4 times generic, its subtitle says three times, and the paper's excerpt attaches 2.4x to a different pair: human over LLM. I would quote "almost triples", from the abstract, and treat 2.4x as a probable mix-up.
The cost finding is the one I would act on. At about $0.03 per email and a 10% click rate, the personalisation bill is about $0.30 per click ($0.03 divided by 0.10). That counts personalisation only, not infrastructure or delivery. It is still cheap enough to change who gets targeted.
Weinz: the same label, opposite results
Weinz and colleagues ran simulations at 3 organisations in 2024, with free-tier GPT-3.5 Turbo and open-source profiling. Across about 18,000 emails per arm, LLM mail did worse: 5.5% of opened emails led to a page visit, against 8.0% for a traditional button email.
The firm-level split is the real lesson. At the mid-sized company, 31.5% of opened LLM emails led to a visit, against 3.9% for the button email. At the company with over 30,000 staff, it was 4.5% against 8.1%. The method won by 8 times in one place and lost by nearly half in another. Any single-organisation result, including a vendor's case study, could land anywhere in that range.
The lure moves AI phishing click rates more than the author does
The clearest pattern across these tests is that context dominates authorship. Within Bethany's human-written arm, the supervisor lure drew 21.67% and the remote-job lure 4.47%, a ratio of 4.8. In the LLM arm, it was 21.34% and 4.84%, a ratio of 4.4. On those same lures, swapping the author barely moved the rate.
A separate randomised trial with no AI arm supports the point. Ho and colleagues sent 10 phishing campaigns to more than 19,500 UC San Diego Health employees over 8 months. Click rates ran from 1.82% for an Outlook password lure to 30.8% for a vacation policy lure. That is a 17-fold range from the lure alone, in one workforce, with no model involved.
So when a chart shows one AI email beating one human email, you have learned little until you see several lures. The 1-email-per-arm designs in this table cannot separate the writer from the pretext.
What phishing simulation vendors report from their platforms
Phishing simulation vendors publish real measurements, but the incentives and disclosure are different. Each number below comes from a company that sells phishing defence or training. That does not make them false. It means you should ask what is missing, using the same discipline the 88% versus 37% AI ROI survey gap calls for.
Hoxhunt: a 24% lead that is 0.53 points
Hoxhunt says its AI agent beat elite human red teams by 24% in March 2025. The underlying failure rates were 2.78% against 2.25%. That is 1 extra click per roughly 190 emails. The page gives no human arm size, no interval and no significance test, and it states the lead three ways: 24%, 23.8% and 23.3%.
Four months earlier, the humans were ahead, 2.3% to 2.1%. Hoxhunt's own text warns that the 2023 baseline is not like for like. Osterman Research adds that every user was trained on Hoxhunt's platform, so the cohort is self-selected. I would call this a credible signal of narrowing, and not evidence that AI has passed people.
IBM and SoSafe: humans ahead, with thin designs
IBM X-Force sent one human email and one ChatGPT email to 800 healthcare staff each in 2023. Humans drew 14% and AI 11%, and staff reported the AI email more often. The widely quoted build times were 5 minutes against 16 hours, though IBM's Stephanie Carruthers told SecurityWeek she spent hours working out which prompts worked.
SoSafe's 2023 data showed 21% clicks on AI email and 27% on human email, with equal open rates of 78%. Its chief executive noted that the AI emails were not personalised. Both tests used models that are now several generations old, a point the next section takes seriously.
| Claim | What it rests on | Our rating |
|---|---|---|
| AI phishing is 4.5x more effective than human phishing | 54 vs 12, which matches AI against a control arm in a 101-person study | Misread. Not an AI vs human ratio if the match holds. |
| AI now writes phishing as well as skilled people | Bethany, Heiding 2024, partly Hoxhunt | Supported for typical attackers, less so against the best human spear phishing. |
| AI has overtaken human red teams | Hoxhunt, 0.53 points, undisclosed human sample | Marketing until the method is published. |
| Personalisation at scale raises clicks | Czybik, 7,741 participants, peer reviewed | Best evidence in this set. |
| AI multiplies attacker profit by up to 50x | A model in Heiding 2024, with assumed inputs including marketing conversion rates | Estimate. The authors call their estimates likely overstated. |
Where this argument is weakest
Almost every model tested here is old, and that is the strongest objection to my framing. GPT-3.5, early GPT-4 and small local models do not represent 2026 capability. Czybik's authors argue their result is a floor, because they used simple local models on forewarned, trained people. If you read the 2023 to 2025 trend as narrowing, near-term parity could become an AI lead, and these studies would miss it.
Parity is not comfort
A reader could accept every number here and still be right to worry. If AI matches a competent human at a few cents per email, attackers can personalise for targets who were never worth the human hours. Measured click rate per email is the wrong unit for that risk. Total successful compromises would be better, and no study here measures them. The FBI complaint data in our review of deepfake CFO fraud controls shows the losses are large, without showing how much AI contributed.
Limits of this review
This is not a systematic review. I selected 8 tests I could trace, without a pre-registered protocol. Simulations also measure something narrower than attacks, because security filters may treat simulated mail differently and some clicks could be automated scanners. Several papers do not describe bot filtering in the parts we read.
Two evidence gaps deserve naming. We could not parse the Czybik PDF, so its arm-level rates come from excerpts. The generic rate appears as both 3.7% and 3.9% there. We also could not parse Microsoft's PDF, so the link to Heiding remains an inference. Both can be checked in minutes by someone with the documents, and I would welcome the correction.
What to change in your phishing simulation programme
Change the threat model before you change the vendor. The evidence supports planning for more personalised email at lower attacker cost. It does not support buying anything on the strength of a 4.5x claim. In practice, this means testing your programme with lures built from what is publicly linkable about your own staff, because that variable has the best evidence behind it.
Do not expect training alone to carry this. Ho and colleagues found embedded training cut the likelihood of clicking by about 2%, and found no significant relationship between annual training and failure. That trial had no AI arm, so it cannot speak to AI email directly. I would still move marginal budget toward controls that hold after the click, including the payment checks covered in our comparison of BEC protection plans.
Then use these questions on any vendor that shows you an AI phishing number.
| Ask | A good answer | A warning sign |
|---|---|---|
| What was the comparison arm? | Named: human expert, generic template or no email | "Traditional" or "non-AI", undefined |
| How many people per arm? | Counts for every arm, with intervals | A platform user total, or simulations for one arm only |
| How many different lures? | Several, with results shown per lure | One email per arm |
| What counts as a click, and over what denominator? | Sent or opened, stated, with scanner clicks removed | A rate with no denominator |
| Was the gap in points or percent? | Both, with absolute rates shown | A relative lift on its own |
Frequently asked questions
Is AI phishing more effective than human phishing?
Not on the controlled evidence so far. In 8 head-to-head tests, pooled AI results ranged from 39 points behind human writers to 1.2 points ahead. The largest same-pool test found 12.15% for LLM email and 10.92% for human email across 9,129 recipients. Skilled human spear phishing still leads in several tests. AI clearly beats generic templates, and it does so at very low cost.
What is the AI generated phishing click rate?
There is no single rate, because results depend on the lure and audience. Controlled tests range from 2.78% in a heavily trained vendor population to 54% among about 25 paid university volunteers. In the 7,741-person USENIX field experiment, LLM personalised email reached 10.0%. In a university workforce of 9,129, the rate was 12.15%. Compare rates only inside one study, never across studies.
Where does the 54% AI phishing click rate come from?
Microsoft's Digital Defense Report 2025 is the usual citation, but the figures match a 2024 study by Heiding and colleagues. That study had 101 participants, with 54% for AI, 54% for human experts and 12% for a control. It also cites a 50-times profitability figure, as Microsoft does. We could not confirm the footnote in Microsoft's report, so treat this as a strong match.
Does LLM spear phishing work better than generic phishing?
Yes, and this is the best-supported finding here. Czybik and colleagues tested about 7,700 people at USENIX Security 2026 and report that LLM personalisation almost triples clicks over generic phishing, at about $0.03 per email. Manually crafted spear phishing still did better, at 24.2% against 10.0%. The participants knew they were in a study and had security training.
Are phishing simulation vendor statistics reliable?
They are real measurements with thin disclosure, so treat them as signals. Hoxhunt's 24% lead for AI is 2.78% against 2.25%, with no published human arm size or significance test. IBM and SoSafe both found humans ahead, in older tests with limited designs. Ask each vendor for the comparison arm, the count per arm, the number of lures and the click denominator before you rely on a figure.
Does security awareness training stop AI social engineering?
The evidence is weak and mostly not AI specific. A randomised trial of more than 19,500 UC San Diego Health employees found embedded training cut the likelihood of clicking by about 2%, with no AI arm included. Hoxhunt reports its trained users fail at low rates, but it has no outside comparison cohort. Treat training as one layer, and protect payments and credentials after the click.
Where to start this week
Pull the last AI phishing statistic that appeared in a board deck or vendor proposal at your company. Run it through the 5 questions above. If nobody can name the comparison arm or the arm size, take the figure out of the deck, and replace it with your own click rate per lure from your last 3 simulations.
Next, ask your simulation owner to run one test this quarter that uses public information about your staff, with a generic version sent to a matched group. A few hundred recipients per arm will not settle the question, but it will tell you whether personalisation moves your own numbers.
If a click turns into a wire
Click rates are the top of the funnel. For what happens when the money has already left, read how the wire recall window shapes BEC recovery.
References
- Heiding, Lermen, Kao, Schneier and Vishwanath, Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns, arXiv, 30 November 2024. A later version appears in Expert Systems with Applications, volume 314, June 2026. Retrieved 6 November 2026.
- Heiding, Schneier, Vishwanath, Bernstein and Park, Devising and Detecting Phishing: Large Language Models vs. Smaller Human Models, arXiv, version of 30 November 2023. Retrieved 6 November 2026.
- Bethany et al., Lateral Phishing With Large Language Models: A Large Organization Comparative Study, IEEE Access 13, 2025. Retrieved 6 November 2026.
- Czybik, Kouam, Heubl, Nold and Rieck, A Large-Scale Study of Personalized Phishing using Large Language Models, USENIX Security, August 2026, and the BIFOLD release, 11 August 2026. Retrieved 6 November 2026.
- Weinz, Zannone, Allodi and Apruzzese, The Impact of Emerging Phishing Threats, ACM ASIA CCS, August 2025. Retrieved 6 November 2026.
- Microsoft figures: The Register, 16 October 2025; Microsoft Source EMEA, 5 November 2025; Cloud Security Alliance research note, 14 June 2026. Secondary reporting of the report.
- Vendor tests: SecurityWeek on IBM X-Force, 24 October 2023; Hoxhunt, April 2025; Osterman Research, 14 May 2025; SoSafe release as republished, April 2023; it-daily on the SoSafe sample.
- Ho et al., Understanding the Efficacy of Phishing Training in Practice, IEEE Symposium on Security and Privacy, 2025, via the UC San Diego release, 16 September 2025.
Weakest thing about this source base: we read the Czybik and Microsoft figures through excerpts and secondary reports, because neither PDF would parse, and three of the 8 tests are vendor-run with incomplete methods. Differences, intervals and per-email-sent rates are Zan Digital calculations from published counts. Figures current as of 6 November 2026.
Related reading