robots.txt Checker — is your page or its resources blocked?
Your `robots.txt` file is the first thing a crawler reads — and a single `Disallow` line can hide a page from Google and AI engines entirely. This check confirms the file is reachable at your site root and that no rule blocks the URL you care about or the CSS, JS and images it needs to render. A blocked page can't be crawled, so it can't rank or be cited.
Updated June 2026 · Technical SEO & GEO · Part of Crawlability & Indexability
Check if robots.txt is blocking your page
Paste a URL — GEObubbly fetches your robots.txt and tells you whether the page or its resources are blocked from crawlers.
✓ Free check✓ 1 signal✓ No signup required
In short: Your robots.txt must be reachable at /robots.txt, return a 200, and not contain a Disallow rule that blocks the page or the CSS/JS it needs to render.
What does the robots.txt check verify?
It confirms two things: that robots.txt is reachable (a 200 response at /robots.txt), and that no Disallow rule blocks the URL being audited or the critical resources it loads. Common problems it catches include:
A blanket Disallow: / left over from a staging site.
A rule that blocks a whole folder the page lives in.
Blocking /wp-content/, /assets/ or similar, which stops CSS and JS loading and breaks rendering.
A robots.txt that returns a 404 or 500 (defaults to allow, but worth knowing).
How is it evaluated, and how is it scored?
GEObubbly fetches /robots.txt, parses the User-agent groups and Disallow/Allow rules, and checks them against the audited URL and its key resources. It carries a weight of 3 points in the Crawlability & Indexability category, because a blocking rule here stops everything downstream — the page is never read in the first place.
Why robots.txt matters for SEO and GEO
A crawler that can't fetch a page can't index, rank or cite it. robots.txt is also where you control AI-engine access — the same file that allows or blocks GPTBot, ClaudeBot and PerplexityBot. An accidental Disallow is one of the most common and most damaging SEO mistakes precisely because it's a single line that's easy to copy from a staging environment. Blocking resource folders is subtler but just as harmful: if CSS and JS are blocked, engines can't render the page properly and may misjudge its content. Keep robots.txt minimal and deliberate, and pair this with the GEO / LLM Readiness AI-bot access check.
How this check scores
Pass:reachable, returns 200, path is crawlable.
Warning:no robots.txt at all (crawlers default to "allow," but you lose control).
Fail:a Disallow rule blocks this page or its critical resources.
FAQ
robots.txt is a plain-text file at the root of your domain (yourdomain.com/robots.txt) that tells web crawlers which parts of your site they may or may not request. Each block names a User-agent and lists Disallow and Allow rules. It's the first file most crawlers fetch, so a mistake here can quietly hide pages from search engines and AI answer engines before they ever read a word of your content.