Most failures in this category are invisible from inside your app. A request from GPTBot or PerplexityBot is rejected by Cloudflare, Akamai or AWS WAF before it ever hits your origin — so it never shows up in your application logs, and a normal browser test from your office passes cleanly. The page looks fine to you and is unreachable to the engines that matter.
Below, each check is explained in plain terms: what it is, why it matters, and what counts as a pass, a warning, or a fail. Every check has a stable ID (H1–H9). The first two are core checks (TLS health and edge AI-bot handling); the rest are extended checks covering DNS, delivery, headers, firewall friction, domain trust and consent.
Certificates have fixed lifespans and must be renewed. An unnoticed expiry can take the whole site offline for every visitor and crawler at once, because modern browsers refuse to load sites behind invalid certificates and show a full-page security interstitial most people will not click past.
The same hard failure happens with an untrusted chain (self-signed or an unrecognised authority) or a domain mismatch. Automated renewal (Let's Encrypt, your CDN's built-in renewal) plus independent expiry monitoring is what reliably prevents this.
This is the single most common 'allowed but never cited' trap. Cloudflare Bot Fight Mode, Akamai Bot Manager and AWS WAF managed rule groups frequently classify AI crawlers as "unverified bots" and return a 403 (or a JavaScript challenge they can't solve) before the request reaches your server. Because the rejection happens at the edge, it never appears in application logs.
The fix is configuration, not code: on Cloudflare, explicitly allow the AI user-agents under Settings → Bots → AI Crawlers; on Akamai or AWS WAF, add an allow-list rule that runs before the default bot challenge. Blocking AI crawlers is a valid business decision — but it should be a choice, not an accident.
Extended checks
Diagnostic checks flagged for review — useful polish, but they don't carry the same score weight as the core checks above.
The deliberate version is cloaking — a serious guidelines violation — but the common version is accidental: edge rules, paywalls or JavaScript fallbacks that hand crawlers a reduced or blocked version of the page.
Keep your primary content server-rendered and consistent regardless of user-agent. Don't branch the body on bot detection, and verify CDN/WAF layers aren't quietly degrading what crawlers receive.
A single nameserver or provider is a single point of failure. Major DNS-provider outages have repeatedly taken every site relying solely on them offline at once.
Use a managed DNS provider with global anycast (Cloudflare, Route 53, NS1), publish multiple NS records, and keep TTLs in a sane range so you can move quickly during an incident.
Compression dramatically cuts the transfer size of HTML, CSS and JS — especially Brotli. HTTP/2 multiplexes resources over a single connection; HTTP/3 adds QUIC's faster, more resilient connection setup.
Together they reduce transfer size and connection overhead, improving Core Web Vitals and making crawlers (including AI crawlers) cheaper to satisfy per visit.
HSTS forces browsers to always use HTTPS; CSP restricts what resources a page can load; X-Content-Type-Options and Referrer-Policy plug common browser-level holes.
X-Robots-Tag applies robots directives at the server level — crucial for non-HTML resources like PDFs and images where you can't add a meta robots tag.
Crawlers don't solve CAPTCHAs and don't execute most challenge pages — so a blanket challenge effectively hides content from them. Aggressive rate limits can throttle normal crawl activity.
Allowlist verified search and AI crawlers, keep challenges on interactive endpoints (login, forms, search), and let read-only public HTML through.
SPF lists who may send mail for your domain; DKIM signs messages so receivers can verify authenticity; DMARC ties them together with a policy for failures.
A domain with email authentication, a clean reputation and a history free of spoofing is a well-managed domain — exactly the kind of trust profile both search and AI engines reward indirectly.
If your primary content is hidden or unloaded until consent is given, search and AI crawlers see only the banner, a blank page or a blocked state — and index that instead of your real content.
Implement consent so it governs tracking and cookies, not access to content. Keep the body present in server-rendered HTML regardless of the banner. Balance this with your legal obligations, which vary by jurisdiction.
Why infrastructure, bots & DNS matter for SEO and GEO
This category is uniquely punishing because every failure mode is invisible from inside your stack. You won't find a bot-mitigation 403 in your application logs, and a slow DNS lookup from one region won't show up in your local browser. By the time you notice the symptom — missing AI citations, dropped rankings, traffic that just isn't arriving — the engine has long since moved on.
The fixes, though, are mostly configuration changes at your edge rather than application code: an allow-list rule, a one-click HTTP/2 toggle, a renewed certificate, an enabled AAAA record, a consent banner that doesn't gate content. Low effort, high certainty, often unblocking weeks of otherwise-mysterious traffic loss.
- Reach. If
GPTBot,ClaudeBotorPerplexityBotis blocked at the edge, no amount of great content, structured data or llms.txt will get you cited — they never see the page. - Uptime. TLS, DNS and consent failures don't degrade your site, they remove it — for users and crawlers alike. A single unrenewed certificate or a hard cookie wall can erase visibility overnight.
AI crawlers are stricter and less forgiving than Googlebot: they tend not to retry aggressively, they don't fetch missing TLS intermediates that browsers paper over, they don't run JavaScript-only consent flows, and they're more likely to be classified as 'unverified' by default bot-mitigation rules. A site that's perfectly visible to Google can be completely invisible to ChatGPT and Perplexity because of a single edge rule. Treat AI-bot access (H2) as a tier-one production check — it's the cheapest possible way to lose LLM visibility.
Audit your page across all 9 checks in Infrastructure, Bots & DNS
See exactly which checks pass, warn or fail — in seconds.
Run a free audit →FAQ
GPTBot, ClaudeBot and PerplexityBot as unverified bots and return a 403 or a JS challenge — none of which shows up in your application logs. Explicitly allow these user-agents in your edge configuration (see H2).