Will my noindex actually work?
Every robots.txt tester tells you whether a URL is blocked. None of them tells you the thing that traps people: whether a noindex on that page could ever be read.
It cannot, if robots.txt forbids fetching the page. The instruction is on the page, and the page is never opened. That is why a blocked URL sits in the results for months looking correctly configured.
How it works
The method is stated so the answer can be argued with, which is the only reason to trust a tool you did not write.
- Nothing you type leaves this page
- The check runs in your browser. There is no request, so there is nothing to log, nothing to store and nothing on our side that ever sees your file. You can confirm that in the network tab: pressing the button makes no request at all.
- It fetches nothing either
- You paste the file rather than giving us a URL to fetch. A tool that fetches a URL on your behalf makes a request to an address a stranger chose, which is a liability on a public page with no account behind it. Pasting removes it, and it also means this works on a robots.txt that is not published yet.
- The parser is the one our crawler obeys
- Not a second implementation written for a marketing page. Two copies agree until somebody fixes a bug in one of them, and then the tool explains a decision the crawler did not make. It carries its own tests.
- Longest match wins, and a tie goes to Allow
- That is the rule in the specification, and it is why the result shows which rule decided and what it beat. A verdict with no working is a verdict you cannot check.
- It answers for one crawler at a time
- robots.txt groups rules by user-agent, and a site can say different things to different crawlers. Change the crawler field and the answer can change with it.
What it does not tell you
It reads robots.txt and nothing else. It does not know whether the page actually has a noindex on it, whether Google has already indexed it, or whether the page exists. It answers one question: if you put a noindex there, would anything ever read it.
For the order to do things in once you have the answer, and for why the obvious fix makes the problem permanent, the guide has the whole of it.