robots.txt generator
Pick what to block and it writes the file. Then it reads that file back and tells you what it actually does, which is a different question and the one people get wrong.
The parser doing the reading is the one this product’s own crawler obeys, so the answer is not a second opinion about robots.txt. It is the same opinion.
It runs entirely in your browser. Nothing you type is sent to us, stored, or logged.
Block the usual places
Your robots.txt
User-agent: * Disallow: /admin Sitemap: https://example.com/sitemap.xml
What that file actually does
Every path above, read back through the same parser this product’s crawler obeys. A generator that only hands you text is asking you to take its word for it.
/admin
Blocked, as asked — by Disallow: /admin
Why a generator should check its own output
robots.txt looks like a list and behaves like a precedence system.three things catch people, and none of them is visible in the text a generator hands over:
- The LONGEST matching rule wins, not the first. Order in the file decides nothing, which is the opposite of how a list normally reads.
- A tie goes to Allow. Writing the same path in both boxes does not block it twice as hard; it permits it.
- Groups are per crawler. Rules written for one agent say nothing about another, and an agent reads only the most specific group that matches it.
So the file above comes with its own reading. If a path you asked to block is not blocked, that is reported rather than left for a crawl a month from now to discover.
How to add your sitemap to robots.txt
One line, written as Sitemap: followed by the sitemap’s full address. Google’s instruction is short:
Insert the following line anywhere in your robots.txt file, specifying the path to your sitemap. We will find it the next time we crawl your robots.txt file:
Sitemap: https://example.com/my_sitemap.xml
The questions people ask about it are answered in Google’s specification, and the answers are more relaxed than most guesses:
- Can it be a relative path? No. “It must be a fully qualified URL, including the protocol and host, and doesn't have to be URL-encoded.” The generator above warns when a line is not a full URL.
- Does it go inside a User-agent group? It can go anywhere. “The sitemap field isn't tied to any specific user agent and may be followed by all crawlers, provided it isn't disallowed for crawling.”
- Can I list more than one? Yes, one line each, with no limit on how many. The sitemap can also sit on another host.
Source for all three: How Google interprets the robots.txt specification. To see which fields inside the sitemap Google reads, use the sitemap checker.
What this does not check
- Whether the file is reachable once you upload it. It must sit at the root of the host it governs, and a robots.txt served from a subdirectory governs nothing.
- Whether blocking a page removes it from search. It does not. Blocking stops a page being read, and a page nobody may read can still be listed from what others say about it.
- What other crawlers do. These rules are read the way Google reads them, and an agent that ignores robots.txt entirely is not affected by anything on this page.
- An llms.txt file. That is a reading list for AI agents with no rules in it, and Google Search does not use it.
- Whether you should block any of it. This writes what you choose. Choosing is the part that needs somebody who knows the site.