robots.txt: disallow all
Two lines, and everyone gets them right. What almost nobody carries across is that Google prints two warnings beside that snippet on its own page, and both of them change what you should expect to happen.
It stops crawling, not listing
This is the warning Google puts directly under the snippet, and it is the reason people come back a month later wondering why their pages are still there:
Keep in mind that in some situations URLs from the site may still be indexed, even if they haven’t been crawled.
Blocking a page stops it being read. A page nobody may read can still be listed from what other pages say about it, which is how a bare address with no description ends up in the results.
So if the goal is to disappear from search rather than to save crawl requests, this snippet is the wrong instrument and using it makes the problem permanent, because the instruction that would remove the page is on the page nobody may now fetch.
The crawler it does not stop
The second warning, on the same page, and the one nobody quotes at all:
This doesn’t match the various AdsBot crawlers, which must be named explicitly.
A wildcard that is not actually a wildcard is worth knowing about. If you wrote User-agent: * expecting silence and are still seeing crawl traffic, this is a likely reason, and the fix is to name the agent rather than to write the star more emphatically.
How to allow all crawlers
The opposite mistake, and the gap between the two is a single colon’s worth of text. An empty Disallow does not block nothing-in-particular; it permits everything:
It is functionally equivalent to having no robots.txt file at all, or using an Allow: / rule.
So there are three ways to allow every crawler everywhere, and Google reads them the same: Disallow: with nothing after it, Allow: /, or no robots.txt file. The file’s address has to answer properly for the last one to hold. Google’s specification says “Google’s crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn’t exist.” A server error is read the other way: “For the first 12 hours, Google stops crawling the site but keeps trying to fetch the robots.txt file.”
If you meant to allow everything and crawling stopped, fetch your own /robots.txt and look at the status code before you look at the rules. The How Google interprets the robots.txt specification page lists what happens after those first 12 hours.
Four snippets, and what each does
Block every crawler from the whole site
User-agent: * Disallow: /
The one everybody means. It stops crawling. It does not empty you out of the index, and it does not reach AdsBot.
Allow every crawler everywhere
User-agent: * Disallow:
The empty Disallow is the opposite of the one above, and the difference is a single character. Google says it is the same as having no file at all.
Block one crawler and nobody else
User-agent: SomeBot Disallow: /
Groups are per agent. A crawler you have not named reads the * group instead, and is unaffected by this one.
Block everything except one section
User-agent: * Disallow: / Allow: /public
The longer rule wins, so /public survives. Order in the file has nothing to do with it, which is the part most often assumed.
Disallow one directory, and the slash that decides what it blocks
Blocking a single folder is the other thing people come here for. Google’s instruction is one sentence:
Append a forward slash to the directory name to disallow crawling of a whole directory.
A rule matches any path that starts with it. So the slash is the difference between blocking a folder and blocking every address that begins with the same letters. Each verdict below comes from the parser behind the tester on this site, and a test fails if the page and the parser ever disagree.
With the slash
User-agent: * Disallow: /private/
- /private/ blockedthe directory itself
- /private/report.html blockedanything inside it
- /private allowedthe address without the slash is a different path, and it is not blocked
- /private-offers alloweda neighbour that only shares the letters
Without the slash
User-agent: * Disallow: /private
- /private blockedthe bare address
- /private/report.html blockedthe directory as well
- /private-offers blockedand every path that merely starts with those letters
- /Private/report.html allowedmatching is case-sensitive
The usual accident is the version without the slash: it quietly takes out neighbours that were never meant to go. The version with the slash has the opposite gap, and leaves the bare folder address crawlable, which matters when that address serves a page of its own.
Rather than trusting any of these against your own file, paste it into the tester below with the addresses you care about. It says which rule decided each one and what that rule beat, which is where the surprises usually are.
Blocking AI crawlers works the same way, with a group for each crawler’s user agent. Adding an llms.txt file does not block anything, and the llms.txt guide explains why.