Indexed, though blocked by robots.txt

What robots.txt is actually for

It controls what a crawler may FETCH. It says nothing about what may be listed. Here is Google putting it in one line:

This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.

So a blocked page can still be listed. Google finds the address from somewhere else, usually a link on another site, and it has an address it is not allowed to open. It lists what it has: the URL, and whatever the pages linking to it said about it.

Which is why a result for a blocked page often has no description under it. There is nothing to write one from.

Why adding noindex does not work

This is the part that turns a small annoyance into a page that sits in the results for months. The instinctive fix is to add a noindex to the page while leaving the robots.txt block in place. It cannot work, and Google says so directly:

For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.

Read the mechanism rather than the rule. The noindex lives ON the page. Being told not to fetch the page means never reading anything on it, including the instruction you just added. The two settings do not stack; the first one prevents the second from ever being seen.

So the page stays listed, the noindex sits there looking like a fix, and every subsequent check finds a page that is correctly configured and still in the results.

What to do, in order

The order is the whole of it. These steps done in the wrong sequence produce exactly the situation described above.

1. 1Decide which thing you actually want. There are two different wishes here and they need opposite settings: keeping a page OUT OF THE RESULTS, or keeping Google from spending crawl requests on it. robots.txt does the second. It does not do the first.
2. 2If you want it out of the results, unblock it in robots.txt first. That feels backwards and it is the whole trick: Google has to be allowed to fetch the page in order to read the instruction telling it to drop the page.
3. 3Then put the noindex on the page itself, as a meta tag or an X-Robots-Tag header. Now the next crawl reads it and the page leaves the index.
4. 4Wait for that crawl before doing anything else. Nothing you do at this stage is instant, and the common mistake is to re-block the page a day later, which restores the original problem exactly.
5. 5Only re-block in robots.txt if you genuinely need to save the crawl requests, and only once the page has actually dropped out. For most sites that last step is not worth doing at all.

When it is not a problem at all

Often it is not worth touching. This status is a report, not an error, and Search Console lists it under the things it wants you to look at rather than the things it says are broken.

If the addresses involved are search pages, filters, carts, or the endless query-string variations a shop generates, then a listing nobody will ever click is a smaller problem than the crawl requests you were trying to avoid. Leaving the block in place and accepting the odd bare URL in the results is a legitimate answer.

The case that does deserve action is a page that should never have been public: a staging copy, an internal tool, a document with somebody’s name on it. For those the answer is not noindex at all. It is a password, because robots.txt was never access control and a listed URL is an invitation to try it.

What this product records about it

ShownTrue obeys the rules a site publishes about what may be visited, so a page you have blocked is a page it does not fetch either. What it keeps is the record: which addresses it found, which it was not allowed to open, and which of the pages it did read ask not to be indexed.

That distinction is the useful one here, because a page that was never fetched and a page that was fetched and said nothing are different situations that look identical in a list of addresses. Recording the difference is what lets the two be told apart later.

The crawler works, is tested, and has crawled this site over the real internet. No customer has ever run one, so there are no results on this page about anybody’s site. The capability pages state exactly what exists and what does not.