A file nobody asked you for is suddenly on everyone's checklist.
You've probably seen it in an audit, red, with a warning: "No llms.txt found." It looks like a missing smoke alarm. It isn't. Here's what the file is, who it was really built for, and why we tell our customers to spend that afternoon on something else.
What an llms.txt file is
An llms.txt file is a short Markdown file you put on your website, usually at yoursite.com/llms.txt, that gives AI tools a tidy summary of the site and a list of links to the pages that matter. The idea comes from a public proposal at llmstxt.org, first written in 2024 and now in a second version.
The proposal's reasoning is fair. Web pages are built for people, wrapped in menus, ads and scripts, and turning them back into clean text is fiddly. So, it says, give the machines a clean version: a title, a one-paragraph summary in a quote block, then sections of links, ideally to Markdown copies of your pages. Only the title is required.
It is not robots.txt. Robots.txt tells crawlers where they may go. An llms.txt file tells a reader what's worth reading. One is a gate. The other is a brochure.
Who it's really for
Read the proposal and the answer is in plain sight. It says llms.txt files "are used most heavily for software documentation, where coding agents follow them to find API references and tutorials." The AI companies themselves publish them for their own developer docs.
That is a real use. A programmer's assistant looking up how a library works is helped by a clean map of the docs. If you run developer documentation, the file may well earn its keep.
A plumber's website is not developer documentation.
What Google has said
In April 2025, Google's John Mueller answered a question about llms.txt on Reddit, as reported by Search Engine Journal: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag."
The keywords meta tag is the old tag where sites listed what they claimed to be about. Search engines stopped trusting it years ago, because a claim about a page is weaker than the page itself. Mueller's point was the same: why read what a site says about itself when you can read the site?
Google's own guidance on its AI features, updated in December 2025, is plainer still: "You don't need to create new machine readable files, AI text files, or markup to appear in these features."
That's the company that runs AI Overviews and AI Mode, telling you in writing that the file isn't needed for them.
What we do with it
Our website checks note whether /llms.txt exists, and mark the line "informational." We never count it as a failure, we never put it in your recommendations, and a missing file never costs you anything in our report. When there's none, the line says so plainly: few sites have one and its effect is not proven.
We'd rather you spent the time on what does move answers. Let AI search crawlers read your site. Keep your key text in the page, not behind JavaScript. Keep your name, address and phone the same everywhere. Be on the sites AI cites for your kind of business. Those are the things we test, and they're the things Google's guidance lists.
Adding an llms.txt file won't hurt you. It also won't get you named. Don't let anyone bill you for it as if it would.
The file is a summary of your site written for machines. The machines would rather read your site.
Questions
What is an llms.txt file for?
It gives AI tools a short Markdown summary of a site and a list of links to its most useful pages. The proposal at llmstxt.org says it's used most heavily for software documentation, where coding agents use it to find API references. For a business website, Google has said no AI text files are needed to appear in its AI features.
Should I use txt or md?
The proposal names the file `llms.txt`, but its contents are Markdown, and it suggests offering Markdown copies of pages at the same address with `.md` added. So: the file is `.txt`, the writing inside is Markdown, and the linked pages can be `.md`. If you're a business site rather than a docs site, you can skip all three.
Why do we use a robots.txt file?
To tell crawlers which parts of your site they may and may not visit. It's a request, not a lock: the standard says plainly that its rules "are not a form of access authorization." The AI companies' crawlers, such as OAI-SearchBot, ClaudeBot and PerplexityBot, say they follow it. Our free crawler check reads yours and tells you which AI crawlers it lets in.
Is AI scraping illegal?
We're not lawyers, and the answer depends on where you are and what's done with the content, so get legal advice for your case. What we can say is technical: robots.txt is a request crawlers are asked to honour, not access control. Major AI companies publish their crawler names and say they respect it, and services such as Cloudflare now let sites block AI crawlers by default.
Sources
| Claim | Source | Checked on |
|---|---|---|
llms.txt proposal: Markdown file at /llms.txt (or a subpath); H1 is the only required section, then a blockquote summary and H2 "file lists"; Markdown copies of pages at .md; first written in 2024, now v2; "used most heavily for software documentation, where coding agents follow them…"; AI labs (OpenAI, Anthropic, Gemini) publish llms.txt for their developer docs |
llmstxt.org: The /llms.txt file | 15 Nov 2026 |
| John Mueller quote ("AFAIK none of the AI services have said they're using LLMs.TXT … comparable to the keywords meta tag"), as reported 17 Apr 2025 | Search Engine Journal: Google says LLMs.txt comparable to keywords meta tag | 10 Nov 2026 |
| Google: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." (page last updated 10 Dec 2025) | Google Search Central: AI features and your website | 15 Nov 2026 |
| robots.txt rules "are not a form of access authorization" (RFC 9309, Sept 2022) | RFC 9309: Robots Exclusion Protocol | 10 Nov 2026 |
| OpenAI, Anthropic and Perplexity publish crawler names and say they follow robots.txt | OpenAI: Overview of OpenAI crawlers; Anthropic: Does Anthropic crawl data from the web?; Perplexity: Perplexity crawlers | 10 Nov 2026 |
| Cloudflare changed its default to block AI crawlers (blog post, 1 Jul 2025) | Cloudflare blog: Content Independence Day (1 Jul 2025) | 10 Nov 2026 |
| ScanFree's check: "An llms.txt file exists (informational)"; missing-file text "Few sites have one and its effect is not proven."; never a recommendation | ScanFree: Our own results | 12 Nov 2026 |