H3 · Infrastructure, Bots & DNS

User-Agent Consistency Checker — do bots see the same content?

**If your site serves AI crawlers different content than it shows real users, engines may cite something your visitors never see — or miss your content entirely.** This check looks at whether the content served to bot user-agents matches what a normal browser receives. Differences can come from intentional cloaking, but more often from bot-detection or rendering setups that quietly strip content for crawlers.

What does the user-agent consistency check look for?

It compares what your site serves to different user-agents to confirm bots and users see the same thing. Specifically:

- Content parity — whether the main content served to crawler user-agents matches what a regular browser receives.

- Stripped or altered content — whether bots get a reduced, blocked or different version of the page.

- Cloaking signals — deliberate or accidental serving of different content based on the user-agent.

Consistent content across user-agents passes; minor differences are a warning; substantially different or stripped content for bots is a fail.

How is it evaluated, and how is it scored?

GEObubbly requests the page as different user-agents and compares the content returned. It's an extended Infrastructure check that runs server-side, since it compares responses across user-agents.

Criteria: Pass — same content to all agents. Warning — minor differences. Fail — bots blocked or served different content.

Why content consistency across user-agents matters for GEO

Engines should see the same content your users see. When a site serves different content based on the user-agent, problems follow. The deliberate version is cloaking — showing crawlers one thing and users another — which is a serious guidelines violation. But the more common version is accidental: bot-detection systems, edge rules or rendering setups that, often unintentionally, serve crawlers a stripped-down, blocked or fallback version of the page. Either way, the consequence for GEO is real: if AI crawlers receive reduced or different content, engines either cite something your actual visitors never see, or fail to pick up your real content at all. This compounds the JavaScript-rendering issue — a crawler that gets a JavaScript-stripped fallback effectively sees a different page than the user's browser renders. The fix is to ensure your server-rendered content is consistent regardless of user-agent: don't tailor the actual content to bots, and make sure bot-detection or edge layers aren't quietly degrading what crawlers receive.

How this check scores

  • Pass: Same content delivered to all user-agents.
  • Warning: Minor differences between bot and browser content.
  • Fail: Bots blocked or served substantially different content.

FAQ

Why would a site serve different content to bots?

Sometimes deliberately — which is cloaking, a guidelines violation — but far more often accidentally. Bot-detection systems, CDN edge rules, paywalls, or rendering setups can serve crawlers a different version of a page than users get: a stripped-down fallback, a challenge page, a JavaScript-free shell, or blocked content. This can happen without anyone intending it, as a side effect of security or performance configurations that treat bot traffic differently. The result is that engines receive content that doesn't match what real visitors see, which is why checking for parity across user-agents matters even when no deception is intended.

What is the difference between this and cloaking?

Cloaking is the deliberate, deceptive practice of showing search engines different content than users to manipulate rankings, and it's a clear guidelines violation. Content inconsistency across user-agents is the broader phenomenon of bots and users receiving different content — which includes intentional cloaking but also unintentional cases caused by bot detection, edge rules or rendering fallbacks. So all cloaking is content inconsistency, but not all content inconsistency is cloaking; much of it is accidental. Both are problems for GEO because they mean engines may cite or miss content that differs from what users actually see.

How does inconsistent content hurt AI citation?

If AI crawlers receive a different or stripped-down version of your page than users see, two bad outcomes follow. Engines might cite content based on what the crawler got — which could be a fallback or reduced version your visitors never encounter, producing inaccurate citations — or the crawler might get so little usable content that your real material isn't picked up at all. Either way, the gap between what bots and users see undermines accurate representation in AI answers. Ensuring crawlers receive the same full content as users is what keeps your citations faithful to your actual pages.

How do I make sure crawlers get the same content as users?

Serve your content consistently regardless of user-agent: don't branch your actual page content based on whether the visitor looks like a bot, and ensure your main content is in the server-rendered HTML so it doesn't depend on client-side rendering that crawlers might not perform. Check that bot-detection, CDN and firewall layers aren't quietly serving crawlers a degraded or challenge version. Testing the page as different user-agents and comparing the results reveals discrepancies. The goal is parity — the same full content for everyone — which keeps both search and AI engines aligned with what your users see.

Is it ever okay to serve different content to bots?

There are legitimate cases of varying the response — for example serving correctly-localised content, or returning appropriate status codes — but the core content of a page shouldn't differ between bots and users in a way that misrepresents it. Practices like dynamic rendering (serving a pre-rendered version to crawlers) are acceptable only when the content is equivalent to what users get, not a different or reduced version. The line is whether the substance is the same: adapting delivery is fine; serving bots materially different content from users is cloaking and a problem for both SEO and GEO.

Audit your page across all 9 checks in Infrastructure, Bots & DNS

Run a free audit →