**Your `robots.txt` file is the first thing a crawler reads — and a single `Disallow` line can hide a page from Google and AI engines entirely.** This check confirms the file is reachable at your site root and that no rule blocks the URL you care about or the CSS, JS and images it needs to render. A blocked page can't be crawled, so it can't rank or be cited.
It confirms two things: that robots.txt is reachable (a 200 response at /robots.txt), and that no Disallow rule blocks the URL being audited or the critical resources it loads. Common problems it catches include:
- A blanket Disallow: / left over from a staging site.
- A rule that blocks a whole folder the page lives in.
- Blocking /wp-content/, /assets/ or similar, which stops CSS and JS loading and breaks rendering.
- A robots.txt that returns a 404 or 500 (defaults to allow, but worth knowing).
GEObubbly fetches /robots.txt, parses the User-agent groups and Disallow/Allow rules, and checks them against the audited URL and its key resources. It carries a weight of 3 points in the Crawlability & Indexability category, because a blocking rule here stops everything downstream — the page is never read in the first place.
A crawler that can't fetch a page can't index, rank or cite it. robots.txt is also where you control AI-engine access — the same file that allows or blocks GPTBot, ClaudeBot and PerplexityBot. An accidental Disallow is one of the most common and most damaging SEO mistakes precisely because it's a single line that's easy to copy from a staging environment. Blocking resource folders is subtler but just as harmful: if CSS and JS are blocked, engines can't render the page properly and may misjudge its content. Keep robots.txt minimal and deliberate, and pair this with the GEO / LLM Readiness AI-bot access check.
robots.txt is a plain-text file at the root of your domain (yourdomain.com/robots.txt) that tells web crawlers which parts of your site they may or may not request. Each block names a User-agent and lists Disallow and Allow rules. It's the first file most crawlers fetch, so a mistake here can quietly hide pages from search engines and AI answer engines before they ever read a word of your content.
Open yourdomain.com/robots.txt and look for any Disallow rule whose path matches your page or the folders it lives in, under either User-agent: * or a specific bot. Also check it isn't blocking resource folders like /assets/ or /wp-content/, which would stop CSS and JS from loading. GEObubbly's checker above does this automatically — paste your URL to see whether the page or its resources are blocked.
Not directly — if robots.txt returns a 404, crawlers default to allowing everything, so the site is still crawlable. It's treated as a mild warning rather than a failure. That said, having a clean robots.txt is good practice: it lets you declare your sitemap location, manage crawl budget on large sites, and explicitly allow or block AI crawlers. A missing file is acceptable; a misconfigured one that blocks important pages is the real risk.
That's a strategic decision. Allowing GPTBot, ClaudeBot, PerplexityBot and Google-Extended is the price of being read and cited by AI answer engines; blocking them protects your content but removes you from those answers. Many sites block AI bots by accident through copied rules, so the key is to decide deliberately. Note that blocking a bot in robots.txt is separate from blocking it at your CDN edge, which this site's Infrastructure checks also cover.
Search engines and some AI crawlers render pages to understand them, and they need the CSS and JavaScript to do that. If robots.txt disallows the folders those files live in, the engine sees a broken, unstyled or empty version of your page and may misjudge its content, layout or mobile-friendliness. Modern guidance is to never block resource files crawlers need for rendering — only block genuinely private or duplicate paths.