A4 · Crawlability & Indexability

    X-Robots-Tag Checker — is a header-level noindex hiding your page?

    The `X-Robots-Tag` is a `noindex` you can't see in your page source — it lives in the HTTP response header, not the HTML. This check inspects that header for any indexing directive (noindex, nofollow, nosnippet and more). Because it's invisible when you "view source," a header-level noindex is one of the hardest indexing problems to spot — and it can silently de-index a page, a folder, or even non-HTML files like PDFs and images.
    Updated June 2026 · Technical SEO & GEO · Part of Crawlability & Indexability

    Check your X-Robots-Tag header

    Paste a URL — GEObubbly reads the X-Robots-Tag HTTP response header and flags any hidden noindex or snippet limit that's suppressing the page.

    ✓ Free check✓ 1 signal✓ No signup required
    In short: The `X-Robots-Tag` HTTP header carries the same indexing directives as the meta robots tag — but from the server response, so it never shows in the page source. A header `noindex` can de-index pages and files (including PDFs) without any visible trace.

    What does the X-Robots-Tag check look for?

    It inspects the HTTP response header for the X-Robots-Tag and flags any directive that limits indexing or display — the same set the meta robots tag uses, but delivered by the server instead of the HTML:

    • `noindex` — removes the URL from search and AI results entirely (a critical fail).
    • `nofollow` — tells engines not to follow links from the page.
    • `nosnippet` / `max-snippet` — limits the text shown, which also limits what AI engines can quote.
    • `noarchive` — blocks the cached copy.

    Because the header applies to the response, it can carry noindex for non-HTML files too — PDFs, images and downloads that have no place to put a meta tag. No restrictive header is a pass; a header noindex is a fail.

    How is it evaluated, and how is it scored?

    GEObubbly reads the raw HTTP response headers returned for the URL and parses any X-Robots-Tag values, including bot-specific variants (e.g. googlebot: noindex). It's a core check worth 2 points in Crawlability & Indexability, and it runs partially — confirming a header directive reliably requires inspecting the server response, which GEObubbly does during a full audit. It pairs with the meta robots check, because the same noindex can live in either place.

    Why the X-Robots-Tag matters for SEO and GEO

    A header-level noindex is uniquely dangerous because it leaves no visible trace. You can view the page source, see no noindex, and still be completely de-indexed — the directive is in the response header, only visible if you inspect the HTTP headers directly. It's commonly set by a server config, a CDN rule, a security plugin, or a framework default, and it can apply to an entire path at once, quietly removing many URLs. The X-Robots-Tag is also the only way to apply noindex to non-HTML resources, so a misconfigured rule can hide PDFs and images you actually want found. For both search rankings and AI citation, a hidden header noindex has the same effect as the visible meta version: the page never appears. Always check the headers, not just the source — this and the visible meta robots tag together make up the Crawlability & Indexability indexing signals.

    How this check scores
    Pass: no restrictive X-Robots-Tag in the response headers.
    Warning: header carries snippet/follow limits but not noindex.
    Fail: X-Robots-Tag: noindex silently de-indexes the page or file.

    FAQ

    The X-Robots-Tag is an HTTP response header that lets you apply indexing directives — like noindex, nofollow and nosnippet — at the server level instead of inside the HTML. It does exactly what the <meta name="robots"> tag does, but because it travels in the response header rather than the page body, it never appears when you "view source." Search engines and many AI crawlers read it the moment they fetch the page, so a directive here takes effect immediately and invisibly.

    Related checks in Crawlability & Indexability

    A1
    Indexability verdict
    A3
    Meta robots directives
    A2
    robots.txt reachable & not blocking

    Audit your page across all 14 checks in Crawlability & Indexability

    See which checks pass, warn or fail — in seconds.

    Run a free audit →
    ← Back to the full Crawlability & Indexability guide
    Free TrialContact