**The `<meta name="robots">` tag can quietly remove a page from search with one word: `noindex`.** This check reads your meta robots directives and flags anything that stops the page being indexed, followed or snippeted — `noindex`, `nofollow`, `nosnippet`, `noarchive` or a restrictive `max-snippet`. An accidental `noindex` is one of the most common reasons a perfectly good page never appears.
It reads the <meta name="robots"> tag (and engine-specific variants like googlebot) and flags directives that limit indexing or display:
- `noindex` — removes the page from search and AI results entirely (a fail).
- `nofollow` — tells engines not to follow links from the page (a warning).
- `nosnippet` / `max-snippet:0` — prevents a text snippet, which also limits AI quoting.
- `noarchive` — blocks the cached copy.
No restrictive directives is a pass; a noindex is a critical fail.
GEObubbly reads the rendered <head> for the meta robots tag and parses every directive it contains. It's a core check worth 3 points in Crawlability & Indexability. It pairs with the X-Robots-Tag header check, because the same noindex can be set in the HTTP header instead of the HTML — and that version is invisible in the page source.
A noindex is the single most direct way to remove a page from search, and it's frequently left on by accident — a CMS template default, a "discourage search engines" toggle, or markup copied from a staging build. Because the page otherwise looks and works perfectly, nobody notices until traffic quietly disappears. Snippet-limiting directives are subtler but matter for GEO: nosnippet and a low max-snippet restrict the text engines can show — and, by extension, the passages AI answer engines can quote from you. Keep meta robots open (index, follow) on pages you want seen, and only restrict deliberately. See the related Crawlability & Indexability guide for the full picture.
The <meta name="robots"> tag is an HTML directive in the page's <head> that tells search engines how to treat that specific page. Its values control indexing (index/noindex), link following (follow/nofollow) and how the page can be displayed (nosnippet, noarchive, max-snippet). Unlike robots.txt, which controls crawling, the meta robots tag controls what an engine does after it has read the page — most importantly, whether it's allowed to index it at all.
noindex tells search engines to read the page but never show it in results — it removes the page from the index entirely. Use it intentionally on pages you don't want found: thank-you pages, internal search results, thin tag archives, or duplicate utility pages. The danger is using it accidentally: a noindex left on a page you do want ranking will keep it out of search no matter how good the content is, which is exactly what this check is designed to catch.
They solve different problems. Disallow in robots.txt stops a crawler from fetching the page at all; noindex in the meta robots tag lets the crawler fetch and read the page but stops it from being indexed. A subtle trap: if you both Disallow a page and add noindex, the engine may never crawl it, so it never sees the noindex — and could still index the URL from external links. To reliably keep a page out of search, allow crawling and use noindex.
noindex is primarily a search-indexing directive, and AI crawlers treat it inconsistently — some respect it, others focus on robots.txt and their own user-agent rules. What's more reliably relevant for AI visibility is snippet control: nosnippet and a low max-snippet limit the text engines can display and therefore the passages AI answer engines can quote. If you want a page both ranked and citable, keep it index, follow with no snippet restrictions.
Because it may be set in the HTTP response header instead of the HTML. The X-Robots-Tag header can carry the same noindex directive, and it never appears when you view the page source — it's only visible if you inspect the response headers. This makes header-level noindex one of the hardest indexing problems to spot. GEObubbly checks both the meta tag and the X-Robots-Tag header so a hidden directive can't slip through.