X-Robots-Tag
It is the robots meta tag moved into the HTTP response, which means it can carry a noindex on a file that has no HTML to put one in, and it means a page can be carrying one that nobody can see in view-source.
Most people arrive here the second way round: a page will not index and the markup looks fine. A section further down lists where the header comes from when nobody on the team set it.
What it is
The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag.
“Any rule” is the sentence to keep. noindex, nofollow, nosnippet, noarchive, max-snippet, max-image-preview, unavailable_after: whatever you could write in a meta tag, you can send as a header instead. A response may carry several of them, either as repeated headers or as one comma-separated list.
A header may also name a crawler before the rules, so X-Robots-Tag: googlebot: nofollow addresses Google alone. “Rules specified without a user agent are valid for all crawlers. The HTTP header, the user agent name, and the specified values are not case sensitive.”
The two jobs it does that a meta tag cannot
The first is files. A PDF has no head section, so there is nowhere to put a meta tag:
To block indexing of non-HTML resources, such as PDF files, video files, or image files, use the X-Robots-Tag response header instead.
The second is reach. “The benefit of using an X-Robots-Tag with HTTP responses is that you can specify crawling rules that are applied globally across a site.” One rule in a server config can cover every URL matching a pattern, which is the whole appeal and also the reason a stale rule does so much damage: nothing in any page’s source changed.
robots.txt silences it, which is the trap
Putting a Disallow on a URL that also sends a noindex stops the noindex from ever being read:
robots meta tags and X-Robots-Tag HTTP headers are discovered when a URL is crawled. If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored. If indexing or serving rules must be followed, the URLs containing those rules cannot be disallowed from crawling.
The header is sent with the response, so a crawler that is not allowed to make the request never receives it. The same page states the rule once more in its opening paragraphs: “Keep in mind that these settings can be read and followed only if crawlers are allowed to access the pages that include these settings.”
This is what produces a page listed in Google with no description, and indexed, though blocked by robots.txt is the Search Console status for it. If you want a page out of the index, the Disallow has to go so the noindex can be read. The robots.txt tester shows whether a given URL is disallowed by your current file.
Where two robots rules disagree, Google resolves it one way: “In the case of conflicting robots rules, the more restrictive rule applies. For example, if a page has both max-snippet:50 and nosnippet rules, the nosnippet rule will apply.” So a meta tag saying index and a header saying noindex is a noindex.
Where the header comes from when nobody set it
None of these show up in the HTML, so reading the page source will not find them.
The host, on a deployment URL
Vercel sends X-Robots-Tag: noindex on every preview deployment, and on a production deployment once a newer one has been promoted. Netlify sends it on Deploy Previews and on older branch deploys. A link or sitemap entry that points at one of those deployment URLs, instead of your own domain, points at a page carrying a noindex.
A CDN or proxy rule
A rule written once to keep a staging path out of search applies to whatever later matches the same pattern.
A server config block you inherited
An .htaccess or NGINX location rule for PDFs or images, widened at some point to match more than it was written for.
Every one of them is found the same way: request the URL and read the response headers rather than the body. In a terminal that is curl -sI https://example.com/page; in a browser it is the Network tab, selecting the document request and reading its response headers. Search Console’s URL Inspection reports the same finding under Indexing allowed.
Before you spend time on any of that, rule out the case above where neither the header nor the meta tag can be read at all. The noindex checker takes your robots.txt and a URL and answers that one question. It reads robots.txt only; it does not fetch the page, so it cannot tell you what the header says.
Checking it across a whole site
One URL at a time is enough to settle an argument about one page. A header set by a pattern is a different problem, because the only way to know how far the pattern reaches is to ask every URL. ShownTrue’s crawl fetches each page itself and records the robots directives it was sent, headers included.
Its honest state: the crawl reads robots directives from both the meta tag and the header, stores them, and marks a page noindexed by either. It obeys robots.txt on its own fetches. It merges the two sources, so it records that a page is noindexed but not whether the header or the meta tag said so, and working out which rule is responsible is still a person’s job. The capability page says exactly what is built.