What is llms.txt, and does Google use it?
llms.txt is a Markdown file at the root of a site that lists the pages an AI agent should read first. It was proposed in September 2024 and it is still a proposal. Google has said its Search does not use it, in AI Overviews or anywhere else.
It is often mixed up with robots.txt, which is the file that actually controls what crawlers may fetch. If keeping a crawler out is what you came here for, that is a robots.txt rule, and the ShownTrue robots.txt generator writes one and reads it back to you.
What the file is
The proposal, published by Jeremy Howard on 3 September 2024, describes itself in one line:
A proposal to standardise on using an /llms.txt file to provide information to help agents use a website.
The format is short. A heading with the site’s name is the only part it requires (“An H1 with the name of the project or site. This is the only required section”). After that come an optional one-paragraph summary in a quote block, then lists of links under second-level headings, and by convention a section called Optional for pages an agent can skip when it is short of room. The proposal also suggests publishing a plain Markdown copy of each page at the same address with .md added.
# Example Co > Example Co makes accounting software for small shops. This file lists the pages > that explain the product and its documentation. ## Docs - [Getting started](https://example.com/docs/start.md): install and first invoice - [API reference](https://example.com/docs/api.md): every endpoint, with examples ## Optional - [Changelog](https://example.com/changelog.md)
A made-up company, to show the shape. It lives at /llms.txt, next to robots.txt.
Does Google use llms.txt?
No. Google lists it first among the things you can ignore in its guide to generative AI features in Search:
You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.
The same paragraph heads off the usual counter-argument, that Google has been seen fetching llms.txt files: “Note that Google may discover, crawl, and index many kinds of files in addition to HTML on a website: this doesn't mean that the file is treated in a special way.” Googlebot fetching a file is not the same as Search using it.
That statement covers Google Search. It says nothing about other companies’ assistants, and none of the sources on this page say which of them read the file. Treat any claim that a particular AI product ranks or cites you because of llms.txt as unproven until its maker says so.
Why Lighthouse checks for it
Chrome’s Lighthouse has an llms.txt audit, which has led some people to read it as a requirement. It is not one. The audit fails only when requesting the file causes a server error, and a missing file passes as not applicable:
If the file is not provided by the server (resulting in a 404), the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment.
So the useful check is what /llms.txt returns on your site today. A clean 404 is fine. A 500, which some setups send for any unknown address, is what Lighthouse flags, and it usually means other missing pages error too.
llms.txt vs robots.txt
robots.txt sets rules. llms.txt is a reading list. The proposal says its file is read differently from robots.txt: “llms.txt information is instead used on demand, when an agent needs information”. Nothing in an llms.txt can keep a crawler out of a page.
| Question | robots.txt | llms.txt |
|---|---|---|
| What it is for | Telling crawlers which paths they may fetch | Pointing an agent at the pages worth reading |
| Format | Plain text rules, one per line | Markdown: a title, a summary, lists of links |
| Blocks anything | Yes, for crawlers that obey it | No. It is a list, with no rules in it |
| Read by Google Search | Yes | No, by Google's own statement |
| Standing | A published standard Google documents | A 2024 proposal |
If you want to stop a named AI crawler, find the user agent its operator publishes and give it a group in robots.txt. The robots.txt tester shows whether a rule you wrote blocks the path you meant, and the disallow all guide covers what blocking does to pages Google has already indexed.
Should you add one?
For Google Search, it makes no difference either way, on Google’s own word. The case for one is narrower: a documentation site or a product with an API, where people point AI coding tools at your docs and a short index of the right pages saves them from reading the wrong ones.
If you add one, keep it short and keep it true. Every link in it should answer 200, and it should change when the pages it lists change. A stale reading list is worse than none, because it sends an agent to pages that no longer say what it expects.
This site has no llms.txt, for the reason in the Google quotation above. Its robots.txt and sitemap are what search engines read.