Crawlability and indexability are the foundation of the entire audit. We weight other categories more heavily for GEO, but this one is a prerequisite: a page that can't be crawled or indexed scores zero everywhere that matters, because it simply won't appear. Think of these nine checks as the locks on the front door — if any are jammed, the quality of the rooms inside is irrelevant.
Below, each check is explained in plain terms: what it is, why it matters, and what counts as a healthy result. Every check has a stable ID (A1–A9) so you can map a finding in your report straight to the explanation.
In short: The indexability verdict combines HTTP status, robots.txt, meta robots, X-Robots-Tag and canonical into one yes/no answer. If any signal blocks indexing, the page can't appear in search or be cited by AI.
Indexability is not one setting; it is the combination of several. A page can pass eight checks and still be invisible because one signal quietly says "do not index." The indexability verdict exists so you don't have to hold all of those signals in your head at once — it resolves them together the way a crawler does.
The logic mirrors how Google and AI engines actually decide: first, did the URL return a usable 200? Then, is crawling allowed by robots.txt? Then, do the on-page meta robots tag and the X-Robots-Tag header avoid noindex? Finally, does the canonical point to this page (or to a different, indexable one)? If every layer is clear, the page is eligible.
This roll-up is why a page that "looks fine" can still be missing from search. Treat the verdict as the headline and the individual checks below as the diagnosis.
Pass: every signal clear — the page is eligible for indexing.
Warning: indexable, but a soft signal is off (e.g. nofollow).
Fail: a noindex/block from any source makes the page non-indexable.
In short: Your robots.txt must be reachable at /robots.txt, return a 200, and not contain a Disallow rule that blocks the page or the CSS/JS it needs to render.
robots.txt is the first file a crawler requests — a plain-text set of allow/disallow rules per user-agent. Get it wrong and you can wall off your entire site with a single line: Disallow: /. This is one of the most common catastrophic SEO mistakes — usually a staging rule that shipped to production.
Two subtleties matter. First, blocking a page in robots.txt does not reliably keep it out of the index — Google can still index a blocked URL from external links, just without its content, producing an ugly "no information is available" snippet. If you want a page out, use noindex, not Disallow. Second, never block the CSS and JavaScript a page needs to render; if a crawler can't fetch those resources it may misjudge the page's layout and mobile-friendliness.
For AI visibility, robots.txt is also where you allow or block AI crawlers like GPTBot, ClaudeBot and PerplexityBot.
Pass: reachable, returns 200, path is crawlable.
Warning: no robots.txt at all (crawlers default to "allow," but you lose control).
Fail: a Disallow rule blocks this page or its critical resources.
In short: The meta robots tag in the page head controls indexing and snippet behavior. The directive that matters most is noindex — if it is present, the page will be removed from search results.
The <meta name="robots"> tag lives in the <head> and speaks directly to indexing engines. Key values: noindex (drop from index), nofollow (don't pass link equity), nosnippet, noarchive, and max-snippet.
A noindex is the cleanest way to keep a page out of search — but also the most common way pages disappear by accident. Thin pages, filtered listings, internal search results and staging templates often ship with a noindex that nobody removes when the page becomes important. Because the tag is invisible to users, the only way to catch it is to inspect the rendered head.
Snippet directives like nosnippet and a low max-snippet are a GEO concern too: they limit how much of your text search and AI surfaces can quote, directly reducing your chance of being cited.
Pass: no restrictive directives.
Warning: snippet limits (nosnippet, low max-snippet) or nofollow only.
Fail: noindex present — excluded from search and AI surfaces.
In short: The `X-Robots-Tag` HTTP header carries the same indexing directives as the meta robots tag — but from the server response, so it never shows in the page source. A header `noindex` can de-index pages and files (including PDFs) without any visible trace.
Everything the meta robots tag can do, the X-Robots-Tag HTTP header can do too — and it has one capability the meta tag lacks: it can control indexing of non-HTML files like PDFs and images, which have no <head> to put a tag in.
The danger is visibility. You will not see an X-Robots-Tag by viewing source — it lives in the response headers, set by the server, a CDN, or a security plugin. A page can look perfectly indexable in its HTML while a header silently says noindex.
Pass: no restrictive X-Robots-Tag in the response headers.
Warning: header carries snippet/follow limits but not noindex.
Fail: X-Robots-Tag: noindex silently de-indexes the page or file.
In short: A healthy page returns a 200 status with real content. Anything else — a 3xx redirect, a 4xx error, a 5xx fault, or a soft 404 — means crawlers get something other than the page you intended.
The HTTP status code is the server's one-word verdict on a request. 200 means "here is the content." 3xx means "it moved." 4xx means "it's not here" (404) or "you can't have it" (403). 5xx means "the server broke." Crawlers need stable 200s to index a page; repeated errors erode crawl frequency and trust.
The sneaky failure is the soft 404: a page that returns 200 OK but actually shows "page not found" or empty content. A genuine missing page should return a true 404 (or 410 if permanently gone), not a 200.
Always evaluate the final status after redirects.
Pass: 200 OK with real, substantive content.
Warning: a 3xx that resolves to 200, or a suspected soft-404.
Fail: a 4xx or 5xx — the page is broken or missing.
In short: Redirects should be direct and minimal: ideally none, or a single clean 301 to the final URL. Long chains and loops waste crawl budget, slow the page, and can cause crawlers to give up before they reach your content.
Redirects are necessary — for moved pages, http→https, www normalization — but each hop adds latency and consumes crawl budget. A chain like http → https → www → trailing-slash makes a crawler do four round-trips for one page.
Two anti-patterns to fix: redirect chains (three or more hops — collapse them so the first URL points straight to the destination) and redirect loops (A→B→A, which never resolves and is fatal). Prefer 301 (permanent) over 302 (temporary) for moves you intend to keep.
AI fetchers are even less patient than search crawlers — a slow or multi-hop URL is more likely to be dropped before the content is read.
Pass: direct 200, or a single clean 301 to the final URL.
Warning: two hops, or a 302 used for a permanent move.
Fail: three or more hops, a loop, or a chain that ends in an error.
In short: A canonical tag tells engines which URL is the master version of a page. It must point to an absolute, indexable, 200 URL — and for your primary pages, it should usually point to itself.
The same content is often reachable at many URLs: with and without a trailing slash, with tracking parameters, via http and https, via www and non-www. Left unmanaged, engines treat these as separate pages and split your ranking signals. The <link rel="canonical"> tag consolidates them onto one.
Two rules. First, the canonical should be absolute and point to a page that returns 200 and is itself indexable — canonicalizing to a redirect, a 404, or a noindex page is a common, quiet way to lose rankings. Second, for the primary version of a page, use a self-referencing canonical.
Only point the canonical elsewhere when the page is a genuine duplicate of another URL you want indexed instead. A wrong canonical can remove a perfectly good page from consideration.
Pass: present, absolute, points to a valid, indexable URL (self-referencing for the primary version).
Warning: canonical points cross-domain, or to a redirect/non-self where unexpected.
Fail: canonical points to a broken/blocked URL — or is missing where duplicates clearly exist.
In short: An XML sitemap is a machine-readable list of the URLs you want crawled. It should be reachable, return a 200, and be well-formed XML — a urlset or a sitemap index.
A sitemap doesn't guarantee indexing, but it dramatically helps discovery, especially for large sites, brand-new pages, and pages that aren't well linked internally. It tells crawlers "these are the URLs I consider important, and here's when they last changed."
The file must be valid XML — a <urlset> of URLs, or a <sitemapindex> for big sites. Common failures: a sitemap that 404s, one full of noindex or redirected URLs, or one that was never regenerated after a migration and now lists dead URLs.
Keep sitemaps clean: only canonical, indexable, 200 URLs belong in them.
Pass: reachable, returns 200, valid urlset/sitemapindex XML.
Warning: exists but malformed, partial, or lists non-indexable URLs.
Fail: no sitemap found at any common location.
In short: Even a perfect sitemap helps less if crawlers can't find it. Declaring it with a Sitemap: line in robots.txt is the standard way to make it discoverable to every engine.
You can submit a sitemap in Google Search Console and Bing Webmaster Tools, but those are per-engine. The universal method is a single line in robots.txt: Sitemap: https://example.com/sitemap.xml. Every compliant crawler reads robots.txt first, so this guarantees discovery without per-engine setup.
It costs one line yet is frequently missing — teams generate a sitemap but never advertise it, slowing discovery of new and updated content.
If you use multiple sitemaps, declare the sitemap index URL here.
Pass: a Sitemap: directive is present in robots.txt.
Fail: not declared — the sitemap is discoverable only by guessing the URL.
Why this category is the foundation
It's tempting to jump straight to content and structured data — the things that feel like "optimization." But crawlability and indexability sit upstream of all of it. The sequence an engine follows is strict: discover the URL → crawl it (allowed by robots.txt, clean 200, no redirect chain) → render and read it → index it (permitted by meta robots, X-Robots-Tag, sane canonical) → rank or cite it. Steps 1–4 are this category. A noindex left on a template, a staging Disallow: /, a canonical pointing at a dead URL — any one silently removes a page from the running, and because the failures are invisible to humans, they can persist for months.
The AI-era twistAI crawlers (GPTBot, ClaudeBot, PerplexityBot) are even less forgiving than classic search bots. They rarely execute JavaScript, they're quicker to abandon slow or redirect-heavy URLs, and many sites unknowingly block them at the CDN edge while their robots.txt says "allow." Clean crawlability is now the price of admission to being cited in AI answers, not just ranked in search.