What does the AI bot access check look for?
It fetches your robots.txt and checks whether any rule blocks the major AI crawlers, either individually or via a blanket Disallow: / under User-agent: *. The user-agents it looks for include:
- `GPTBot` & `OAI-SearchBot` — OpenAI (training and ChatGPT Search).
- `ClaudeBot` — Anthropic (Claude).
- `PerplexityBot` — Perplexity.
- `Google-Extended` — Google's AI products (Gemini, AI Overviews training).
- `CCBot` — Common Crawl, which feeds many models.
No major answer-engine bot blocked is a pass; one or two blocked (e.g. a deliberate training opt-out) is a warning; the major answer-engine bots blocked is a fail.
How is it evaluated, and how is it scored?
GEObubbly fetches /robots.txt, parses the User-agent groups and Disallow/Allow rules, and checks each AI user-agent against them. It's the heaviest single check in GEO / LLM Readiness — worth 5 points — because access is the first hurdle: if an AI crawler can't fetch your pages, none of the other GEO signals can help you.
Why AI bot access matters for GEO
Answer engines build their responses from content they're allowed to crawl. Block their user-agents in robots.txt and you remove yourself from that pool — you can't be read, learned from, or cited by ChatGPT, Claude, Perplexity or Google AI Overviews, even if you rank perfectly in classic search. Many sites do this unintentionally: a copied robots.txt, an over-broad security default, or a "block AI scrapers" plugin that quietly disallows the very bots that power citations. It's worth deciding deliberately — and noting that opting out of training (e.g. blocking Google-Extended) is a different decision from blocking the bots that fetch content for live answers. One more subtlety: robots.txt is only the first gate. Your CDN or WAF can silently return a 403 to AI bots at the edge even when robots.txt says "allow", so allowing them here is necessary but not always sufficient. Getting AI access right is the entry ticket to the rest of GEO / LLM Readiness.