**The `X-Robots-Tag` is a `noindex` you can't see in your page source — it lives in the HTTP response header, not the HTML.** This check inspects that header for any indexing directive (`noindex`, `nofollow`, `nosnippet` and more). Because it's invisible when you "view source," a header-level `noindex` is one of the hardest indexing problems to spot — and it can silently de-index a page, a folder, or even non-HTML files like PDFs and images.
It inspects the HTTP response header for the X-Robots-Tag and flags any directive that limits indexing or display — the same set the meta robots tag uses, but delivered by the server instead of the HTML:
- `noindex` — removes the URL from search and AI results entirely (a critical fail).
- `nofollow` — tells engines not to follow links from the page.
- `nosnippet` / `max-snippet` — limits the text shown, which also limits what AI engines can quote.
- `noarchive` — blocks the cached copy.
Because the header applies to the response, it can carry noindex for non-HTML files too — PDFs, images and downloads that have no place to put a meta tag. No restrictive header is a pass; a header noindex is a fail.
GEObubbly reads the raw HTTP response headers returned for the URL and parses any X-Robots-Tag values, including bot-specific variants (e.g. googlebot: noindex). It's a core check worth 2 points in Crawlability & Indexability, and it runs partially — confirming a header directive reliably requires inspecting the server response, which GEObubbly does during a full audit. It pairs with the meta robots check, because the same noindex can live in either place.
A header-level noindex is uniquely dangerous because it leaves no visible trace. You can view the page source, see no noindex, and still be completely de-indexed — the directive is in the response header, only visible if you inspect the HTTP headers directly. It's commonly set by a server config, a CDN rule, a security plugin, or a framework default, and it can apply to an entire path at once, quietly removing many URLs. The X-Robots-Tag is also the only way to apply noindex to non-HTML resources, so a misconfigured rule can hide PDFs and images you actually want found. For both search rankings and AI citation, a hidden header noindex has the same effect as the visible meta version: the page never appears. Always check the headers, not just the source — this and the visible meta robots tag together make up the Crawlability & Indexability indexing signals.
The X-Robots-Tag is an HTTP response header that lets you apply indexing directives — like noindex, nofollow and nosnippet — at the server level instead of inside the HTML. It does exactly what the <meta name="robots"> tag does, but because it travels in the response header rather than the page body, it never appears when you "view source." Search engines and many AI crawlers read it the moment they fetch the page, so a directive here takes effect immediately and invisibly.
They carry the same directives but live in different places. The <meta name="robots"> tag sits in the HTML <head> and is visible in the page source; the X-Robots-Tag is part of the HTTP response header and is invisible unless you inspect the headers. Two practical differences follow: the header version can be applied to non-HTML files (PDFs, images) that have no <head>, and it can be set server-wide for whole folders at once. If both are present, engines honour the most restrictive directive.
You can't see it by viewing the page source — you have to inspect the HTTP response headers. Use your browser's developer tools (Network tab → click the document → Response Headers) and look for an X-Robots-Tag entry, or run a header-checking tool. GEObubbly's checker above does this for you: paste your URL and it reads the response header and flags any hidden noindex or snippet limit, so you don't have to dig through headers manually.
Almost always because the noindex is in the X-Robots-Tag response header rather than the HTML. The page source looks perfectly clean, but the server is telling crawlers not to index it. These header directives are commonly added by server configuration, a CDN or WAF rule, a security plugin, or a CMS/framework default — often applied to a whole path, so several pages disappear at once. Checking the response headers is the only way to catch it, which is exactly what this check is for.
Yes — and that's one of its most important uses. PDFs, images and other non-HTML files have no <head> to hold a meta robots tag, so the X-Robots-Tag header is the only way to control their indexing. That cuts both ways: it lets you intentionally keep private documents out of search, but a misconfigured server rule can also noindex PDFs and images you want found, removing them from search and image results without any obvious sign. If important files aren't appearing, check their response headers.