**If your robots.txt blocks the AI crawlers, you've opted out of being read and cited by AI answer engines entirely.** This check inspects your `robots.txt` for rules that disallow the major AI user-agents — `GPTBot`, `OAI-SearchBot`, `ClaudeBot`, `PerplexityBot`, `Google-Extended` and `CCBot`. Blocking them is sometimes deliberate, but it's just as often an accident — and it quietly removes you from ChatGPT, Claude, Perplexity and Google AI Overviews.
It fetches your robots.txt and checks whether any rule blocks the major AI crawlers, either individually or via a blanket Disallow: / under User-agent: *. The user-agents it looks for include:
- `GPTBot` & `OAI-SearchBot` — OpenAI (training and ChatGPT Search).
- `ClaudeBot` — Anthropic (Claude).
- `PerplexityBot` — Perplexity.
- `Google-Extended` — Google's AI products (Gemini, AI Overviews training).
- `CCBot` — Common Crawl, which feeds many models.
No major answer-engine bot blocked is a pass; one or two blocked (e.g. a deliberate training opt-out) is a warning; the major answer-engine bots blocked is a fail.
GEObubbly fetches /robots.txt, parses the User-agent groups and Disallow/Allow rules, and checks each AI user-agent against them. It's the heaviest single check in GEO / LLM Readiness — worth 5 points — because access is the first hurdle: if an AI crawler can't fetch your pages, none of the other GEO signals can help you.
Answer engines build their responses from content they're allowed to crawl. Block their user-agents in robots.txt and you remove yourself from that pool — you can't be read, learned from, or cited by ChatGPT, Claude, Perplexity or Google AI Overviews, even if you rank perfectly in classic search. Many sites do this unintentionally: a copied robots.txt, an over-broad security default, or a "block AI scrapers" plugin that quietly disallows the very bots that power citations. It's worth deciding deliberately — and noting that opting out of training (e.g. blocking Google-Extended) is a different decision from blocking the bots that fetch content for live answers. One more subtlety: robots.txt is only the first gate. Your CDN or WAF can silently return a 403 to AI bots at the edge even when robots.txt says "allow", so allowing them here is necessary but not always sufficient. Getting AI access right is the entry ticket to the rest of GEO / LLM Readiness.
For the widest reach across today's answer engines, allow GPTBot and OAI-SearchBot (OpenAI / ChatGPT), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google's AI products) and CCBot (Common Crawl, which feeds many models). Allowing these gives your content the best chance of being read and cited in AI answers. Review the list periodically, since new AI crawlers appear regularly and each needs to be permitted explicitly — a rule for one bot doesn't cover the others.
To allow it, simply make sure no rule disallows it. To block it, add a group to your robots.txt: User-agent: GPTBot on one line, then Disallow: / on the next. The same pattern works for ClaudeBot, PerplexityBot and the others — each bot needs its own named group. Be careful with a blanket User-agent: * + Disallow: /, which blocks everything including AI bots. Use the checker above to confirm whether your current rules allow or block each AI crawler.
It's a strategic trade-off. Allowing AI crawlers is the price of being cited and gaining visibility in AI answers — an increasingly important channel. Blocking them protects your content from being used by those engines, but removes you from their answers entirely. Many sites block AI bots by accident, so the key is to decide on purpose. If brand visibility in AI answers matters to you, allow them; if protecting specific content matters more, block deliberately — and remember training opt-out and answer-crawling are separate decisions.
Two common reasons sit beyond robots.txt. First, your CDN or web application firewall (Cloudflare, Akamai and similar) may be silently returning a 403 or a challenge to AI crawlers at the edge — before the request ever reaches your server, so it never appears in your logs. Second, your content may be rendered with JavaScript, which most AI crawlers don't execute, leaving them an empty page. Check both, then confirm your content is answer-first and extractable; allowing the bots is necessary but not always sufficient.
They're separate decisions controlled by different bots. Some crawlers (like Google-Extended, or GPTBot in its training role) gather data used to train models; others (like OAI-SearchBot and PerplexityBot) fetch live content to answer a user's question right now and cite sources. You can opt out of training while still allowing the answer-fetching bots, so you stay citable in live AI answers without contributing to model training. Blocking only the answer bots, by contrast, directly removes you from those answers.