Should You Let AI Crawlers In? What robots.txt Controls, and What It Does Not

Cloudflare measured one AI crawler taking 70,900 pages for every visitor it sent back. That exchange rate is why blocking AI crawlers looks obvious. It is also why the decision is harder than it looks, because robots.txt cannot cleanly separate the crawl that trains a model from the crawl that cites you.

The short answer

Block the training crawlers, allow the retrieval crawlers, and publish a licence pointer. Most B2B software companies lose more from being missing in AI answers than they gain from withholding training data. Publishers who monetise page views face the opposite arithmetic and should block first, then negotiate.

What robots.txt actually controls

robots.txt is a plain text file at the root of your domain. It names user agents and lists the paths each one may or may not fetch. That is the entire mechanism, and it has no authentication, no enforcement and no audit trail.

Everything else in this argument follows from that one design fact. The file is the only lever most site owners have, and it is a weaker lever than the debate around it assumes.

A request, not a control

The file works because large operators choose to honour it, and because being caught ignoring it is reputationally expensive. Compliance is a norm, not a property of the protocol.

Cloudflare published an investigation on 4 August 2025 describing undeclared crawlers that presented as Chrome on macOS. Those crawlers rotated through addresses outside the published ranges and generated 3 to 6 million requests a day across tens of thousands of domains. Cloudflare attributed the activity to Perplexity and de-listed the company as a verified bot. Perplexity disputed the finding publicly.

My position on this is unambiguous. Treat robots.txt as a statement of intent that most large operators respect and a minority do not, and put anything that genuinely must not be copied behind authentication instead.

Three different jobs, one file

The word "crawler" hides three jobs with opposite economics for the site being crawled. Conflating them is the single most common error in this debate.

The first job is training, where pages are fetched to modify model weights. The second is search indexing and grounding, where pages are fetched to build the index an answer engine queries and cites. The third is a live user fetch, where a page is retrieved because a person asked about it seconds ago.

The first returns nothing to you directly. The third is close to a click. A single blanket disallow treats all three the same, which is why blanket policies produce results their owners did not intend.

Token names in column two change without notice and vendors add new ones. Verify each against the vendor's own published crawler documentation before shipping a file, because a mistyped user agent name fails silently and looks identical to a working rule.

Google is the clearest illustration of where the file runs out of road. Google documents 11 common crawler tokens, and only Google-Extended is an AI usage control. Google states plainly that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search".

That sounds like a clean opt out, and it is not. Google-Extended governs training for Gemini models and grounding in Gemini apps and Vertex AI. AI Overviews are assembled from the Search index that Googlebot builds, so the only exit from AI Overviews is an exit from Google Search. What that has done to paid and organic click behaviour is covered in the analysis of paid search click-through under AI Overviews.

The case for blocking, in numbers

The blocking argument is not sentimental. It rests on a measured imbalance between what these systems take and what they return.

The exchange rate is measurably bad

Cloudflare Radar publishes a crawl-to-refer ratio for each AI platform, dividing the pages a platform fetches by the visitors it sends back. In the week to 26 June 2025 the company measured Anthropic at 70,900 pages crawled per referral, with Mistral at the opposite extreme, sending roughly ten referrals for every page it requested.

Two caveats belong in the same breath as that number. Referrals from native mobile apps often carry no referrer header, which inflates every ratio in the table, and the figures move sharply week to week as platforms change how their consumer surfaces link out. Read the ratio as a shape, not as a settled quantity.

The demand side of that ledger is measured more carefully. Pew Research Center analysed 68,879 unique Google searches from 900 United States adults in April 2025. Users clicked a result in 8% of visits where an AI summary appeared, against 15% where none did. In just 1% of visits did anyone click a link inside the summary itself.

Those are the numbers that make blocking feel like the only self-respecting answer. Before acting on them, weigh what the surviving referrals are worth, because assistant traffic behaves differently from search traffic once it lands, as set out in the breakdown of AI referral conversion multiples.

Blocking is the precondition for a licence

You cannot sell access you are already giving away. That is the strategic core of the blocking case, and it explains why the infrastructure layer moved before publishers did.

Cloudflare announced on 1 July 2025 that it would block AI crawlers by default for new sites, on a network the company says carries traffic for about 20% of the web. Condé Nast, Dotdash Meredith, Gannett, Reddit and Ziff Davis were among the named supporters.

The 2026 version goes further. From 15 September 2026, Cloudflare's defaults block mixed-use crawlers on pages carrying advertising, for new customers and all existing free customers. Alongside it sits Pay Per Use, which compensates publishers when their content appears in an AI answer rather than when a bot fetches it, with Ceramic.ai and You.com as launch partners.

Paying on appearance rather than on fetch is the more honest unit, and it is the first pricing model in this argument that puts the charge where the value actually lands. Cloudflare also reports that more than half of AI crawler traffic re-fetches pages that have not changed, which turns the discussion from principle into an origin bandwidth bill.

The case for letting them in

The counter-argument is not that AI companies deserve free content. It is that the block often fails to achieve its purpose while reliably achieving a cost.

You cannot be cited from an index you are not in

The clearest available evidence comes from a July 2026 analysis by cloro.dev covering 1,058 of the web's most-cited domains. Domains disallowing GPTBot showed a median 0.003 ChatGPT citations per Google-organic appearance, against 0.417 for domains that permitted it.

The same study found the effect is provider-specific. Sites blocking PerplexityBot showed a median Perplexity citation propensity of zero, while others sat above one. It also found training crawlers are blocked three to five times more often than retrieval crawlers, which suggests operators already understand the distinction.

Treat that as directional rather than proven. It is correlational, the corpus reflects one vendor's monitored prompt mix, and domains that block differ from domains that do not in ways nobody has controlled for. Whether these citations track search rankings at all is examined in the comparison of ChatGPT citations against Google rank and in the analysis of collapsing overlap between the two.

There is also a timing problem that blocking cannot solve. A disallow line added in 2026 does not remove text from models trained in 2024, and it does not withdraw pages already sitting in public crawl archives that many models ingest. Blocking is forward-looking, always, and the legal route to addressing what was already taken is a separate fight covered in the review of AI training data copyright cases.

For most business software companies there is a blunter point. The content is the marketing. A comparison page or a pricing explainer exists to be found by somebody evaluating tools, and if that evaluation now begins inside an assistant, then absence from the assistant is absence from the shortlist. That shift in the evaluation path is mapped in the piece on how AI reshapes the B2B shortlist.

Why publishers and SaaS companies answer this differently

The two camps are not disagreeing about the facts. They are running the same calculation on different revenue models and arriving, correctly, at opposite answers.

A publisher earns per page view. An AI answer that satisfies the reader substitutes for the visit, so every uncompensated crawl is a direct subtraction from revenue with no offsetting gain. Blocking is the rational move, and holding out is the only negotiating position available.

A software company earns per closed deal. A page is a cost of acquisition rather than an asset that earns on its own, so a brand mention inside an answer carries value even when nobody clicks. The page was never the product.

I would block on a property that sells impressions and allow on a property that sells software. The common mistake is applying one policy across a company that runs both, which is most companies with a blog and a documentation site.

One test on your own numbers

Published ratios describe the average web, and your site is not the average web. The measurement takes an afternoon and it settles the argument internally in a way no external statistic will.

Pull 30 days of server logs and count requests by AI user agent. Then count sessions referred from the assistant domains in your analytics. Divide the first by the second to get your own crawl-to-refer ratio, per bot, and multiply the referral side by your measured value per session.

Expect the referral count to be understated. Assistant traffic frequently arrives with no referrer at all, which is the measurement problem set out in the analysis of dark discovery and attribution. If your referral numbers look implausibly small, that is a reason to widen the measurement, not a reason to block.

The middle path, written as actual directives

The workable policy splits by job, not by company. You are not deciding whether OpenAI is a good actor, you are deciding which of three jobs you are willing to subsidise.

What a working file looks like

The table below is the shape most sites should end up with. It is written as directives rather than as advice, because the abstraction is where people go wrong.

This is a template, not a recommendation for your specific site. Confirm every token against the vendor's current documentation on the day you deploy, and check the file with a robots.txt tester afterwards. The failure mode of a wrong user agent string is silence rather than an error.

The standards that will make this argument obsolete

robots.txt was designed in 1994 to keep crawlers out of infinite calendar pages. It was never meant to express permission for a use case, a price, or a term of years. The current fight is the sound of a file being asked to do a job it was not built for.

Two efforts are trying to replace it. The IETF's AI Preferences working group is drafting a standard vocabulary for exactly this. Its current document, draft-ietf-aipref-vocab, defines two usage categories, train-ai and search, expressed as a structured field such as train-ai=y, search=n. Absence of a value means the preference is unknown rather than granted.

Note the status honestly. That document is on the standards track and was still an Internet-Draft as of August 2026, not a published RFC, and the mechanism for attaching preferences to content is deliberately handled in a separate document. Nothing here is final.

The second effort is Really Simple Licensing, launched in September 2025, which sits on top of that vocabulary. RSL adds what the IETF work explicitly leaves out, namely machine-readable licensing and compensation terms, referenced from robots.txt with a single License line pointing at an XML file.

Add the licence pointer now. It costs an afternoon, it creates a dated record of the terms you asked for, and that record is worth more in a commercial negotiation than in a courtroom. It is worth something in both.

Where this argument is weakest

Every post on this topic picks a side and omits the part that undermines it. Here is what undermines mine.

Where the citation evidence is thin

The relationship between blocking and lost citations rests on one public study, run by a company that sells AI visibility monitoring, using its own prompt corpus. It is correlational and the authors say so. Large publishers block far more than small sites and differ on dozens of other dimensions, any of which could produce the same gap.

No controlled experiment exists. AI platforms do not publish retrieval logs, citation behaviour varies between models and between weeks, and no publisher is going to randomise its own visibility to find out. That evidentiary hole is permanent, and anyone quoting a precise citation penalty is overreaching.

I have also deliberately not quoted a headline figure for what share of the web blocks GPTBot. The published estimates range from single digits to roughly a quarter of top sites, depending on whose crawl and whose definition of "top" you read. Averaging incompatible samples would produce a number that looks authoritative and means nothing.

The strongest argument against my own recommendation is a collective one. If every site allows retrieval crawlers because being cited is individually rational, then no site ever gets paid, and the equilibrium is worse than the one a coordinated block would produce. Blocking has option value that an individual traffic calculation does not capture.

There is a mirror-image weakness in the blocking case. Blocking is forward-looking, does not touch models already trained, does not stop operators who never read the file, and may cost citations while failing to prevent the copying it was aimed at. Both positions have a version where they lose.

Frequently asked questions

Where to start this week

Open your existing robots.txt first. A surprising number of sites are running a rule somebody pasted in 2023 that blocks a retrieval crawler nobody meant to block, and the file has not been read since.

Then get the measurement rather than the opinion. Pull 30 days of server logs, group requests by AI user agent, and put that count next to your assistant referral sessions from the same window. That single table converts a philosophical argument into a number your team can act on, and it takes about two hours.

The third thing is optional and worth doing anyway. Publish a licence file and reference it from robots.txt, so that whatever the standards process settles on, you already have a dated statement of the terms you asked for.

Related reading