From Shubhi K | Product & Market Analysis
llms.txt Adoption Measured: Who Publishes It, and What Google Says It Won't Do
On this page
The number of sites publishing an llms.txt file grew 8.8 times in twelve months, from 4,088 to 36,120 in one tracker of 3 million sites. Almost nobody reads them. Ahrefs found that 97% of the llms.txt files on its 137,210 customer domains drew zero requests in May 2026, and Google now says in writing that Search ignores the file entirely.
Key takeaways
- llms.txt adoption is real but small outside technical sites. Independent censuses of the top 10,000 and top 100,000 domains in September 2026 put it at 8.3% and 8.6%, while Sistrix puts it under 0.005% of all websites.
- Publishing the file does not mean anything reads it. Ahrefs found 97% of roughly 38,000 valid llms.txt files saw no requests in May 2026, and SEO audit tools sent more of the traffic than AI crawlers did.
- Google Search ignores llms.txt, in its own documentation. Since 15 June 2026 its AI optimisation guide says the file will neither harm nor help visibility or rankings, because Search does not use it.
- No major AI crawler policy says it reads llms.txt. OpenAI, Anthropic, Perplexity and Google all govern crawling through robots.txt, while publishing llms.txt files for their own developer documentation.
The short answer. For Google Search, llms.txt does nothing, and Google says so. For AI crawlers, no vendor documents reading it, and server logs show they rarely ask for it. The file earns its hour only if you run developer documentation that coding agents fetch on demand. For a marketing site, skip it and spend the hour elsewhere.
The reader this is written for is the practitioner: the technical marketer or SEO lead whose founder has asked whether the company needs an llms.txt. They have one afternoon to answer with sources.
What llms.txt is, and what it was built to do
llms.txt is a Markdown file served at the root of a site, at /llms.txt. Jeremy Howard of Answer.AI proposed it on 3 September 2024 at llmstxt.org. The format is simple: an H1 with the project name, an optional one-paragraph summary, and H2 sections that list links with short notes.
The intent matters more than the format. The proposal says the file was expected to be useful mainly for inference rather than training. It is used "on demand, when an agent needs information about a topic while assisting a user." The design target was a model reading your docs mid-task, not a crawler indexing your site.
That distinction got lost on the way into marketing. By 2025 the file was being sold as a lever for visibility in AI answers, a job its own author never claimed for it. Most of the confusion about whether llms.txt "works" comes from judging it against the second job instead of the first.
It is also not an access control. robots.txt tells crawlers what they may fetch, and crawler vendors document honouring it. llms.txt tells a model what you would like it to read. If your real question is about blocking or allowing AI bots, that is a robots.txt decision, covered in the decision guide on which AI crawlers to allow in robots.txt.
llms.txt adoption: four counts that disagree by a factor of 5,000
There is no single llms.txt adoption figure, and anyone quoting one without its population is misleading you. Every published count depends on which sites were checked and what counted as a valid file. The spread runs from under 0.005% to 28%.
| Source and date | Population checked | Adoption figure | Stated caveat |
|---|---|---|---|
| Ahrefs, June 2026 | 137,210 domains using Ahrefs Web Analytics | 28% (38,360 domains) | Customers skew technical; treat as an upper bound. |
| Common Crawl, August 2026 | Random sample of hosts it crawls successfully | 11.72% of sampled URLs returned a text body | One crawl; not the same as the web. |
| SEOmator, September 2026 | Cloudflare Radar top 100,000 domains | 8.6% (8,598 domains) | Root path only; one day's snapshot. |
| Rankability, September 2026 | Tranco top 10,000 domains | 8.3% (830 domains) | Unknowns kept in the denominator. |
| Sistrix, as cited by Common Crawl | All websites | Under 0.005% | Primary report not opened for this post. |
The rows are not comparable to each other, and none is wrong on its own terms. A count of SEO-tool customers, a crawl sample and a census of top domains are measuring three different things.
The 28% figure is a ceiling, and Ahrefs says so
The highest llms.txt adoption number comes from Ahrefs' study of 137,210 domains with traffic in May 2026. It found 38,360 of them, or 28%, publishing a valid file. Ahrefs adds the caveat itself. Its customers "skew more technical and SEO-aware than the web at large, so treat the 28% adoption figure as an upper bound."
That caveat matters most for anyone citing the study. The population is, by construction, sites whose owners already pay for SEO tooling. They are exactly the sites most likely to have read a blog post recommending llms.txt.
The 8.8x growth figure is about a subset, not the web
Originality.ai has tracked llms.txt across more than 3 million websites since mid 2025. It recorded 4,088 llms.txt instances in June 2025 and 36,120 by May 2026. Counting the related llms-full.txt and ai.txt files, the total reached 38,980 sites.
The growth is real. The level is still about 1% of the sites it tracks. Originality's own summary is blunt: more sites are adopting the file, "but almost nothing is reading it."
Top-domain censuses converge near 9%
Two independent scans in September 2026 landed close together. SEOmator requested /llms.txt from the Cloudflare Radar top 100,000 domains on 13 September and found 8,598 real files, or 8.6%. Rankability checked the Tranco top 10,000 on 18 September and confirmed 8.3%.
Both studies had to filter out a large false-positive problem. SEOmator found 10,775 domains returning a normal HTML page with a 200 status at that path, a soft 404 that a naive scanner would count as adoption. Treat any adoption figure from a study that does not describe this filter with suspicion.
Who actually requests llms.txt files
Adoption counts tell you who publishes. Server logs tell you whether the effort lands anywhere. Ahrefs is the only large study that measured requests, using bot analytics across its 137,210 domains.
The finding is stark. Of roughly 38,000 domains with a valid file, 97% saw no requests for it at all in May 2026. The remaining 3%, about 1,100 domains, received every llms.txt request Ahrefs measured. Across all of them, that was about 22,000 requests in a month.
SEO tools ask for the file more than AI does
The breakdown of those 22,000 requests is the most useful number in the whole debate. SEO audit tools sent 21.7% of them. General web crawlers sent 13.1%, and technology profiling services sent 11.6%. Much of the traffic to llms.txt is other software checking whether you have one.
AI systems are present but small. AI agents accounted for 10.5% of requests, AI training crawlers 5.3%, and AI retrieval bots 1.1%, or 233 requests. Retrieval bots are the ones that fetch pages to answer a live question, which is the job llms.txt is usually sold on.
Fetched is not the same as read
Ahrefs flags a second limit worth repeating: "'fetched' doesn't mean 'read'." A request in a log proves a bot downloaded the file. It does not prove a model used it to choose a source, and nobody outside the AI companies can see that step.
So every request count is a ceiling on influence, not a floor. Ahrefs says this directly: "Every figure in this study is therefore a ceiling on actual llms.txt consumption." If the ceiling is 97% zero, the typical site's realistic effect on AI citation is smaller still.
What Google has said about llms.txt, in order
Google's position took 14 months to move from forum replies to documentation. Read in sequence, it never changed direction. Only the format got more official.
From a Reddit reply to a documentation note
In April 2025, John Mueller of Google replied to a thread on r/TechSEO. "AFAIK none of the AI services have said they're using LLMs.TXT," he wrote. He added that server logs show "they don't even check for it." He called it "comparable to the keywords meta tag," as Search Engine Journal reported. The reason: it is a site owner's claim about a site that a system could simply check directly.
Later in 2025, Search Engine Journal reported that Gary Illyes and Amir Taboul confirmed at a Search Central Live Deep Dive Asia Pacific event that Google was not pursuing llms.txt. On 15 May 2026, Google published a guide to optimising for its generative AI search features. On 15 June 2026, it added a note logged as "Clarifying guidance on llms.txt files".
That note is now the primary source, and its wording is unambiguous. Google's AI optimisation guide says creating and maintaining the file "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." Its closing advice is also explicit. It lists "unnecessary AI text files (like llms.txt)" among tactics you can ignore.
Chrome audits it, which is a different product making a different claim
The confusion comes from Chrome, not Search. In May 2026, Lighthouse 13.3 added an Agentic Browsing category with an llms.txt audit. Its documentation calls the file an "emerging convention" and says that without it, "agents may spend more time crawling the site."
Read what the audit does, though. It only flags a server error when fetching the file. A missing file returns a 404 and is marked Not Applicable, "as providing the file is optional at the moment." Lighthouse is not scoring you down for lacking llms.txt. It is checking that a file you chose to publish does not break.
These two positions are consistent once you separate the products. Search, which ranks pages, ignores the file. Chrome's tooling, which anticipates browser agents acting for users, treats it as optional plumbing. Anyone telling you "Google now checks llms.txt" is blending the two.
Which AI platforms document reading llms.txt
The fair test is not what Google Search does. It is whether any AI platform that cites web pages says, in its crawler documentation, that it reads llms.txt. We checked the crawler pages of the four companies whose bots matter most for AI visibility.
| Platform | Crawler docs mention reading llms.txt | What governs crawling | Publishes its own llms.txt |
|---|---|---|---|
| OpenAI | No. The bots page links its docs index only | robots.txt for GPTBot and OAI-SearchBot | Yes, for developer docs |
| Anthropic | No | robots.txt for ClaudeBot, Claude-SearchBot, Claude-User | Yes, for developer docs |
| Perplexity | No. The bots page links its docs index only | robots.txt for PerplexityBot | Yes, for developer docs |
| Google Search and Google-Extended | No. Search guide says it is ignored | robots.txt, including the Google-Extended token | Not on Search Central |
The pattern is consistent. OpenAI's bot documentation describes GPTBot, OAI-SearchBot and ChatGPT-User, and points site owners to robots.txt. Its only llms.txt mention is a pointer to its own documentation index. Perplexity's bots page does the same. Anthropic's support article on crawling does not mention the file.
The interesting column is the last one. The companies that decline to say their crawlers read llms.txt all publish one for their developer documentation. That is not hypocrisy. It is the original use case: a coding agent or a developer pasting docs into a model benefits from a clean index. It says nothing about how those companies treat a marketing site.
One more thing the platforms share: user-triggered fetchers often sit outside robots.txt. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says its user fetcher "generally ignores robots.txt rules." These are the agents the llms.txt proposal was written for. None of them commits to reading it either.
Where the case against llms.txt is weakest
Everything above points one way. Here is where it could be wrong, stated as strongly as we can make it.
Agents are the users, and the data under-counts them
AI agents made 10.5% of the llms.txt requests Ahrefs measured, almost ten times the share from retrieval bots. That is the inference-time use the proposal described. If agentic browsing grows the way Chrome's tooling assumes, the file's audience grows with it, and a May 2026 log snapshot will look like a measurement taken too early.
The samples also miss the sites where llms.txt is most likely to work. SEOmator and Rankability checked the root domain only, so a file on docs.example.com counts as absent. Developer documentation, the best use case, often lives exactly there. We think the case against llms.txt for docs sites is weaker than the headline numbers suggest.
The cost is close to zero, so the bar is low
For many sites the file is already generated by their platform. Common Crawl found 68.27% of the 584,107 files it collected were templated, with Wix alone producing 41.34%. SEOmator traced at least 1,561 top-domain files to Shopify and 463 to Yoast. If your CMS writes it for you, the honest cost is a few minutes of review.
Where we still disagree with the "free option" argument is maintenance. A stale index of links you have since moved is worse than none, and the sites most likely to forget a file are the ones that never needed it. Cheap to create is not the same as cheap to keep accurate.
There is also a risk that cuts the other way. Common Crawl found 6.59% of files carrying policy language the specification does not define, and 1,570 naming specific crawlers, with denials that mostly did not match the same sites' robots.txt. It also found 10 files with top-severity prompt-injection content, four judged genuine. A file meant to help models is also a place to manipulate them, which is a reason some platforms may never trust it.
An llms.txt decision for technical marketers, by site type
The right answer depends on what your site is, not on the trend line. Use this table, then move the saved time to work with measured effect.
| Your site | Recommendation | Reason |
|---|---|---|
| Developer documentation, APIs, SDKs | Publish it, and llms-full.txt if your docs tool generates it | Matches the inference-time use case; agents and developers fetch it. |
| Marketing site on Wix, Shopify or Yoast where it is already generated | Leave it, review it once | Near-zero cost, but check links and remove any policy language. |
| B2B marketing site with no file today | Skip it | No documented reader, no Search effect, and no measured AI citation effect. |
| Any site trying to block or allow AI bots | Use robots.txt, not llms.txt | Crawler vendors document robots.txt; llms.txt is not an access control. |
If a vendor or agency is selling llms.txt as an AI visibility fix, ask one question. Which AI platform documents reading it, and where? We have not found one. Treat the answer you get as a test of the rest of their advice.
The same discipline applies to the adjacent tactic. Structured data has a firmer evidence base than llms.txt, and even there the measured effect on AI citation is thin, as the review of schema markup tests on AI citations sets out. The pages that get cited tend to be pages that already rank, which is the finding in the evidence on GEO versus SEO for AI citations.
If you want to know whether any of this affects you, look at your own server logs rather than any study, including the ones above. Filter for requests to /llms.txt by user agent over 30 days. Building that view properly is covered in the guide to building an AI visibility dashboard.
Frequently asked questions
Does Google use llms.txt?
No. Google's AI optimisation guide for Search, updated on 15 June 2026, says llms.txt files are not needed to appear in Google Search, including its generative AI features. It states that maintaining one will neither harm nor help your visibility or rankings, because Google Search ignores them. Chrome's Lighthouse has a separate audit that only checks the file does not return a server error.
How many websites have an llms.txt file?
It depends on the population. Censuses of top domains in September 2026 found 8.6% of the top 100,000 and 8.3% of the top 10,000. Ahrefs found 28% among its own SEO-aware customers and calls that an upper bound. Sistrix, as cited by Common Crawl, puts adoption across all websites at under 0.005%. There is no single correct figure.
Do AI crawlers like GPTBot and ClaudeBot read llms.txt?
Their documentation does not say they do. OpenAI, Anthropic and Perplexity describe their crawlers as governed by robots.txt, and none of their crawler pages says the bots read llms.txt. In Ahrefs' May 2026 logs, AI training crawlers made 5.3% of llms.txt requests, with GPTBot at 4.51% and ClaudeBot at 0.80%. A fetch does not prove the content was used.
Does llms.txt help SEO or AI citations?
There is no published evidence that it does. Google says Search ignores the file. Ahrefs found 97% of valid files drew zero requests in May 2026, and AI retrieval bots made only 233 of about 22,000 requests. No AI platform documents using llms.txt to choose sources. The measured case for it is in developer documentation that coding agents fetch on demand.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which URLs they may fetch, and the major AI vendors document honouring it for their automated crawlers. llms.txt is a Markdown index of content you would like a model to read. It grants or blocks nothing. The llms.txt proposal itself says the two files have different purposes, so use robots.txt for any access decision.
Where to start this afternoon
Open your server logs and count requests to /llms.txt over the last 30 days, grouped by user agent. If you have no file, count the 404s instead, because those are the bots that asked. That one number is better evidence for your site than every study in this post.
Then make the call from the table above and write it down with the date. If the answer is "skip", put a reminder in six months to re-check one thing only: whether any AI platform has added llms.txt to its crawler documentation. That is the event that would change this recommendation, and you will want to have the baseline when it happens.
Related decision
llms.txt is the file you can probably skip. robots.txt is the one you cannot, because it decides which AI crawlers get your pages at all. Read the robots.txt decision for AI crawlers next.
References
- Google Search Central, Optimizing your website for generative AI features on Google Search, mythbusting section, updated 2026, and documentation changelog entry of 15 June 2026. Used for Google's position.
- Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, 15 June 2026. Used for the 28%, 97% and request mix figures.
- Originality.ai, LLMs.txt Tracking Study and Live Dashboard, updated 3 July 2026. Used for the 4,088 to 36,120 growth.
- Common Crawl, A content analysis of llms.txt files from the July 2026 crawl archive, 31 August 2026. Used for file contents, templating and the Sistrix citation.
- SEOmator, llms.txt: We Scanned the Web's Top 100,000 Domains, 13 September 2026; and Rankability, llms.txt Adoption Research Report, 18 September 2026. Used for top-domain adoption.
- Chrome for Developers, Lighthouse llms.txt audit, updated 5 May 2026. Used for what the audit checks.
- Search Engine Journal, Google says llms.txt comparable to keywords meta tag, 17 April 2025. Used for the Mueller quote and the Illyes report.
- Jeremy Howard, The /llms.txt file, published 3 September 2024. Also checked: crawler documentation from OpenAI, Anthropic and Perplexity, October 2026.
The weakest part of this source base is that the only large request-level study comes from one vendor's customer logs for one month. Its population skews technical, and no AI company has published how, or whether, it uses the file.
Related reading