LLM Judge Methodology

Some audit checks are based on rules a parser can verify (a single H1, a meta description present, structured data valid). Others require judgment — does this content actually answer the question, or is it generic boilerplate? We use a calibrated LLM-based judging layer for those.

Confidence layers

• Verified — deterministic checks. Exact, reproducible, no model involved.
• Assessed — judged by an LLM against a versioned rubric, with quoted evidence.
• Self-reported — derived from page metadata or third-party signals.

Every score in your audit is labelled with one of these badges. You always know how a finding was produced.

How a judgment is produced

1. We extract the page's main content (title, headings, lead, key body) up to ~9,000 characters.
2. We hash the content + active rubric version. If we've judged this exact pair before, we reuse the cached result.
3. Otherwise we send the rubric prompt + content to an AI model and require a structured JSON response with score, band, reasoning, and evidence quotes from the page.
4. For high-stakes checks (e.g. "differentiated positioning"), we run the judgment twice with different temperatures. If the two scores differ by 3+ points, we flag the result as variant and average them.
5. Malformed JSON triggers one corrective retry. Persistent failure is logged as a warning, not a blocking error.

Models

Standard checks run on a fast, cost-efficient model. High-stakes checks run on a stronger reasoning model. Both models are accessed through our managed AI gateway — no third-party API keys are required from you, and we never share your page content with model providers for training.

Rubrics & versioning

Each judgment check has a versioned rubric: prompt template + score bands + JSON output schema. New versions start as drafts. They cannot become active until they pass calibration.

Calibration: the MAE < 1.5 gate

We maintain a curated calibration set of pages with human-assigned scores. Before any new rubric version goes live, we re-judge every page in the set with the candidate rubric and compute the Mean Absolute Error (MAE) versus the human scores. A rubric must have MAE < 1.5 on a 0–10 scale to be promoted to active. This bounds the disagreement between our judge and human reviewers.

Disputing a score

If you believe a judged score is wrong, click "Challenge" on the finding in your audit. You'll propose a score and explain why. Our team reviews each challenge. Accepted challenges are added to the calibration set and drive the next rubric version. This is how the system improves over time — with your evidence.

What we don't do

• We don't claim the judge is perfect. We claim it is bounded, transparent, and improvable.
• We don't hide which scores came from the judge — every "Assessed" badge is explicit.
• We don't penalize content the judge didn't see; failures are logged and excluded from your score.

Last updated: April 2026. Questions? Contact us.