robots.txt tester
Paste your file and a list of addresses. It says which are blocked, which rule decided each one, and every rule that rule beat.
That last part is the bit worth having. Most testers answer allowed or blocked, and the answer is only useful if you can see why: robots.txt matching goes by the LONGEST rule, not by the order they appear in the file, and a tie goes to Allow.
It runs entirely in your browser. Nothing you paste is sent to us, stored, or logged, which matters because most people testing a robots.txt are testing one for a site that is not public yet.
What this does not check
Said before you rely on it, because a tool that answers confidently about the wrong question is worse than one that answers nothing.
- Whether the file is actually reachable. This reads what you pasted. A robots.txt that 404s, times out, or is served as HTML is a different problem and this cannot see it.
- Whether Google has read your latest version. Google caches robots.txt, so a rule you added minutes ago may not be in force yet, whatever this says.
- Whether a blocked page is in the index. Those are different questions and the answer is often yes, because blocking a page stops it being read, not listed.
- Rules for a crawler you did not name. Groups are per agent, so a file can permit one and refuse another, and this answers for the one in the box.
The rules this follows
The same parser this product’s own crawler obeys, so what you are shown here is what it would actually do with your file. Longest match wins; a tie between an Allow and a Disallow goes to the Allow; * matches any run of characters and a trailing $ anchors the end; anything not mentioned is allowed.
If you are here because a page you blocked is still turning up in search results, that is not a robots.txt fault and this tool will tell you nothing useful about it. The guide below is the one that answers it.
This reads robots.txt only. An llms.txt file sets no rules a crawler obeys, so there is nothing in it for this tester to check; what llms.txt is, and whether Google uses it covers the difference.