Cloudflare Pay Per Crawl: the 402 Toll and Anthropic's 38,066:1 Ratio

In July 2025, Anthropic's crawlers fetched about 38,066 pages from Cloudflare sites for every visitor Claude sent back. That figure is the business case for Cloudflare Pay Per Crawl, a system that answers AI bots with an HTTP 402 and a price. Fifteen months after launch, the toll booth is built and the payers are still mostly hypothetical.

How Cloudflare pay per crawl works: the HTTP 402 toll

Cloudflare pay per crawl is a paywall written for machines. It launched in private beta on 1 July 2025, and Cloudflare's own documentation still described it as being in closed beta when last updated on 28 July 2026. You request access through a signup form, and existing enterprise customers go through their account executive.

The design reuses HTTP status code 402, Payment Required. The code has sat in the HTTP specification for decades, reserved and almost never used. Cloudflare gave it a job: every request from a registered AI crawler either carries payment intent and gets the page, or it gets a 402 with a price.

Cloudflare sits in the middle as the merchant of record. It records each billable request, aggregates them, charges the crawler and pays the publisher. Neither side builds a billing system, and that is the most useful part of the product.

The reactive path: 402, then a retry

A crawler that asks without declaring a price receives a 402 and a crawler-price header naming the cost in US dollars. It can retry the same request with a crawler-exact-price header that matches. If the request is valid, the page arrives with a 200 and the charge is logged.

Only one price header is allowed per request. Any malformed request gets a 402 back, which means the system fails toward refusal rather than toward free access. I think that is the right default for a payments system.

The proactive path: declare a ceiling

The more efficient route is to send crawler-max-price on the first request. If the publisher's price is at or below that ceiling, the crawler gets a 200 with a crawler-charged header showing what it paid. If the price is higher, it gets the 402 instead.

The crawler is charged the configured price, not its declared maximum. That detail matters, because it means a crawler can set a generous ceiling without overpaying.

Web Bot Auth proves who is asking

A toll only works if the payer cannot pretend to be someone else. Cloudflare requires crawlers to register and to sign every request using HTTP Message Signatures, the standard published as RFC 9421, with an Ed25519 key pair. The crawler hosts its public key in a directory and sends signature-agent, signature-input and signature headers.

This is the less visible half of the launch and, in my view, the more durable one. User-agent strings can be spoofed by anyone. A signed request cannot, and that changes what a site can safely allow for free.

What a publisher sets: allow, charge or block each AI crawler

The publisher's controls are deliberately blunt. The launch post describes a flat, per-request price across the whole site, then a choice for each crawler between three outcomes. Cloudflare's documentation describes the price as set per zone, which in practice means per domain.

Cloudflare's documentation notes that a block set in its WAF or Bot Management products overrides the charge setting. Allow and charge rules run only after those security layers. Publishers can also waive charges for specific crawlers.

The third row is where most real-world use sits today. Cloudflare's launch post concedes that charging a crawler with no billing relationship is the functional equivalent of a network block. The difference is the message it sends: a 403 says no, a 402 says not for free.

A single domain-wide price is also the product's clearest weakness. A news archive, a live price comparison and a terms-of-service page have very different value to a model. Cloudflare's launch post says it expects per-path and per-content-type pricing later. That is a plan, not a feature, and you should price your decision on what ships today.

AI bot blocking on Cloudflare is now the default for new domains

Pay Per Crawl launched alongside a bigger change. On the same day, 1 July 2025, Cloudflare said it would block AI crawlers by default unless they had permission or paid. CEO Matthew Prince branded the day Content Independence Day in a blog post under his name.

The press release is precise about scope. Every new domain signing up to Cloudflare is now asked whether it wants to allow AI crawlers, and in the release's words, every new domain starts with the default of control. Existing sites keep whatever setting they had.

The demand for this was already visible. Cloudflare had offered a one-click block for AI crawlers since September 2024, and more than one million customers had switched it on by July 2025. The default turned an opt-in that a million customers had already chosen into the starting position for everyone new.

Prince's own framing of the problem used older and larger numbers than the Radar series. He wrote that with OpenAI it is 750 times more difficult to get traffic than it was with the Google of old, and with Anthropic 30,000 times more difficult. Those are rhetorical comparisons in an opinion post. The measured ratios below are the numbers to rely on.

If your own site sits behind Cloudflare, check the setting rather than assuming it. The choice between blocking and allowing at the robots.txt level is a separate decision, weighed in the analysis of what robots.txt actually controls for AI crawlers. Pay Per Crawl adds a third option that robots.txt cannot express.

Cloudflare's crawl-to-referral ratio by AI company

The crawl-to-referral ratio is the number that makes the economics legible. It divides how many HTML pages a platform's bots request by how many visitors that platform's products send to the same sites. A ratio of 5:1 is a fair trade by most standards. A ratio of 38,000:1 is extraction.

Cloudflare published the clearest per-company series in August 2025, comparing January and July of that year across its network. The table below reproduces it and adds one column I calculated: referrals returned per 10,000 crawls, which is easier to reason about than a ratio with five digits.

Ratios from Cloudflare Radar as published on 29 August 2025. The last column is 10,000 divided by the July ratio, calculated by Zan Digital. Google's change is Cloudflare's stated figure, which differs slightly from the rounded values shown.

How Cloudflare defines the ratio

Cloudflare's July 2025 methodology post is specific. Crawls are requests from user agents tied to a platform where the response was text/html. Referrals are HTML requests whose Referer header contains a hostname tied to that platform. Training bots and user-triggered bots from the same company are added together.

Google's referrals from its own network are excluded, because Google prefetches pages through speculation rules. And the ratios are relative to the crawlers and referrers in the dataset, not Cloudflare traffic as a whole.

What moved during 2025

Anthropic's ratio fell 86.7% in six months. The likeliest reason is that Claude gained web search with clickable citations during 2025, so it started sending visitors. Cloudflare's earlier one-week snapshot, for 19 to 26 June 2025, had put Anthropic at 70,900:1 and Mistral at 0.1:1, the only platform sending more visitors than it crawled.

Perplexity went the other way. Its ratio rose 256.7% between January and July, from 54.6 to 194.8 crawls per referral. That is still the best of the AI answer engines in the table, but it is the wrong direction for a company whose pitch to publishers is that it sends traffic.

Cloudflare's 2025 Year in Review, published 15 December 2025, gave its headline finding plainly: Anthropic had the highest crawl-to-refer ratio among the leading AI and search platforms. It also described the series as erratic, with Anthropic's ratio reaching as much as 500,000:1 early in the year when referrals were sparse.

What the crawl-to-referral ratio implies for a crawl price

The ratio becomes useful when you read it as a cost. If a crawler takes R pages for every visitor it sends, then each referred visit costs you R page loads of content. A per-crawl price P turns that into a number a publisher can compare with what a visit earns.

Here is the working, with an illustrative price. Cloudflare does not publish typical prices, and no publisher has disclosed one, so the $0.01 below is a round number chosen to make the arithmetic visible. It is not a recommendation or a market rate.

Illustrative calculation by Zan Digital. The $0.01 price is hypothetical. Ratios are Cloudflare's, July 2025. Actual crawl volumes, prices and referral values vary by site.

The same flat price lands very differently on each company. At one cent a page, Google pays five cents per visitor it sends, which it would barely notice. Anthropic, at its July 2025 ratio, would be paying hundreds of dollars per visitor, which is another way of saying it would stop crawling or start negotiating.

That asymmetry is the strongest argument for the product, and I think it is underrated. A flat price is not neutral. It automatically taxes the crawlers that return least, without the publisher having to make a moral judgement about any of them.

It is also why a single price per domain is a poor instrument for discovery-driven sites. A price low enough to keep a high-referral crawler crawling is irrelevant to the extractive one. A price high enough to matter to the extractive one shuts out the one that sends buyers. Your controls need to be per crawler, which is what the allow setting is for.

Who has joined Cloudflare pay per crawl publicly

This is the section with the thinnest public record, and the record is worth stating exactly. Supporting the permission model, enrolling in Pay Per Crawl, and paying through it are three different things. Most named companies have only done the first.

Publishers that backed the permission model

Cloudflare's 1 July 2025 press release listed about 45 publishers, platforms and industry bodies supporting a permission-based model for AI crawling. The list included The Associated Press, The Atlantic, Condé Nast, Dotdash Meredith, Fortune, Gannett Media, TIME, Ziff Davis, Reddit, Pinterest, Quora, Stack Overflow, Webflow, Universal Music Group and the News/Media Alliance.

Support is not enrolment. The release frames these companies as backing the approach, not as confirmed Pay Per Crawl merchants. Quora said it was committed to supporting initiatives like Pay Per Crawl. Stack Overflow's chief executive Prashanth Chandrasekar said that community platforms that fuel LLMs should be compensated for their contributions.

Several of the same names had already signed direct licensing deals with AI labs, which is a separate route to payment and arguably a more lucrative one for large publishers. Those deals are tallied in the ledger of AI content licensing deals. Pay Per Crawl is the route for everyone too small to get a meeting.

The crawler side: experiments, not invoices

On the paying side, the public record is close to empty. In the July 2025 release, ProRata AI described itself as one of the first AI companies to participate in Cloudflare's initiative. No major model lab has been named as a paying customer in any Cloudflare page I opened.

A year later, Cloudflare's own direction shifted. In a 1 July 2026 post it said it was starting to shape Pay Per Crawl into Pay Per Use, running experiments with Ceramic.ai and You.com. It called the work an experiment, and it disclosed no payment amounts.

The pattern is a two-sided marketplace with one side recruited. That is normal at launch. It becomes a problem if it persists, because a toll that nobody pays is just a block with better branding, and a block is something you could already do for free.

Where this argument is weakest

The crawl-to-referral ratio carries most of the weight in this post. It has three known faults, and Cloudflare states the first one itself.

The ratio overstates the gap, by an unknown amount

Referrals are counted from the Referer header. Cloudflare notes that Claude's native app sends no referrer, and believes the same holds for other native apps. A visitor who clicks a citation inside a desktop or mobile app may arrive looking like direct traffic. Cloudflare's own words are that the calculations may overstate the ratios, and it is unclear by how much.

The 2025 figures are also old for a series this volatile. Secondary trackers report sharply lower Anthropic and OpenAI ratios during 2026. But Cloudflare Radar's live page refused automated access when I tried it, and I will not publish a figure I could not open at the source. Treat the July 2025 numbers as the last fully documented reading, not the current one. Anthropic has also publicly disputed Cloudflare's methodology, according to press reports I could not open directly.

The deeper weakness is that the ratio measures traffic, not value. A model that answers a buyer's question correctly, and names your product, may be worth more than a click. A low referral count is not proof of zero benefit. The case that zero-click visibility still pays is argued in the piece on metrics that replace clicks in zero-click search.

What pay per crawl means for a B2B site deciding on AI discovery

I am writing this for a founder or director of a B2B software or services company, because that reader gets the decision wrong most often. Your pages are not the product. They are the shop window. Charging a crawler to look in the window is a strange way to sell what is inside.

For that reader, the default should be allow for search and answer crawlers, and a considered decision on training crawlers. Use Cloudflare's controls to separate them. The ones that cite you are an acquisition channel, and visitors from them convert at rates that are measured in the analysis of AI referral traffic conversion.

The exception is content that is itself the asset. Original research, benchmark datasets, pricing databases and deep documentation are things a model can absorb and reproduce without sending anyone back. That is where charging, or blocking, protects something real. If you run that kind of page, Pay Per Crawl is worth joining the beta for, even if no crawler pays yet, because the 402 records the demand.

Whatever you choose, measure it first. Most sites have no idea which AI crawlers hit them or how often, and your server logs answer that question for free. The method is set out in the guide to reading AI agent traffic in your logs. Your own crawl-to-referral ratio is more useful to you than Cloudflare's network average.

Frequently asked questions

How to decide your crawler policy this week

Start with your logs, not with Cloudflare's averages. Pull 30 days of requests, count hits by AI user agent, and count visits whose referrer is chatgpt.com, claude.ai, perplexity.ai or copilot.microsoft.com. Divide one by the other for each company. You now have your own crawl-to-referral table, and it will tell you which crawlers deserve the allow setting.

Then sort your pages into two piles: pages that exist to bring a buyer to you, and pages that are valuable in themselves. The first pile stays open. For the second pile, request Pay Per Crawl access and set a price. Accept that for now the 402 is mostly a record of who wanted your content and declined to pay for it.

Related reading