LLM-as-judge: grading with a model you don't fully trust
Monday, 9:40: two support leads can review 50 chats a day; the concierge has 75,000. The new judge model says 97% of answers are grounded, but of the 22 bad answers the humans found last week, it passed 15.
You can ask an AI to grade another AI's answers. It's fast and cheap, but only trustworthy once you've checked it against people.
- Problem
The hotel assistant produces 40k free-text answers a day. You need to grade groundedness and helpfulness on thousands of them per release, and humans can review maybe 100.
- Why the simple system fails
Code checks can't judge prose. A naive judge prompt ("rate this answer 1–10") gives 8/10 to almost everything, prefers long answers, flips its verdict when you swap the order of two answers, and nobody knows whether its numbers match what a human would say.
- Engineering concept
Treat the judge as a measuring instrument: give it a narrow rubric with anchored scores, pick pointwise or pairwise for the question you're asking, control for known biases, and measure its agreement with humans before trusting it.
- Technology
Rubric prompts with structured output, pairwise judging with order swapping, judge-human agreement (percent agreement, Cohen's kappa, confusion matrix), sampling and cheap-judge cascades.
- Trade-offs
Cheaper and faster than humans but biased and drifting; pairwise is sensitive but relative; bigger judges agree more with humans but cost more per run.
The problem
2 min- Omar · On-call SREThe judge model says 97% of production answers are grounded in the policy docs.
- Maya · Product managerOur support leads labelled 150 answers and found 22 with invented fees or rules. The judge passed 15 of them.
- Karim · CTOHumans do 50 a day. We have 75,000. We need the judge to work.
The big idea
4 minA judge is a measuring instrument, not an oracle: trust it only as far as you've measured it against humans.
Swapping in a bigger judge first is the wrong instinct: if you don't know how it's wrong, a bigger model may be wrong the same way. Calibrate it like a thermometer in boiling water.
How it works
5 minAn is a model that grades another model's output, with a rubric, on qualities code can't check.
- 1Narrow rubric
A rubric (): one criterion per call, a 1–4 scale with an example per level, evidence (claims and quotes) before the score.
- 2Human labels
Two people label a calibration set of 150 answers. Their own agreement is the ceiling: kappa 0.84.
- 3Measure the judge
Run it on the same 150. Cohen's kappa () is agreement corrected for chance (1 perfect, 0 chance). Watch the cell "humans say bad, judge says fine": that's what ships bugs.
- 4Fix and repeat
Rewrite the rubric where it disagrees; try a judge from another model family; re-run until kappa nears the human ceiling.
- 5Control biases
Position bias (): preferring the answer shown first or second. Judge pairs in both orders. Tell it length isn't a criterion.
- 6Pin and version
Pin the judge model and rubric version, and print both next to every score.
Watch a decision being made
4 minLina and you calibrate the groundedness judge
Turn v1 (passes 15 of 22 bad answers) into a judge you can trust to block a release.
- 1Read the confusion matrix
"Agreement is 89%. Now look at where."
textjudge: pass judge: fail human: pass 126 2 human: fail 15 7 <- the cell that ships bugsWhy: With 128 of 150 grounded, agreement is inflated. The judge-pass/human-fail cell is the regressions you'd ship.
- 2Compute kappa
Chance agreement is already ≈ 0.81, so kappa ≈ 0.40: weak.
Why: Kappa removes the agreement a judge gets by always saying "grounded".
- 3Fix the rubric
"All 15 misses contain a plausible fee or time not in the sources." New rubric: list claims, quote evidence for each, anchored 1–4, "plausible but absent = NOT FOUND". A judge from another model family.
Why: Most disagreements live in an edge case the rubric didn't define.
- 4Re-measure and pin
v3: 19 of 22 caught, 4 false alarms, kappa ≈ 0.82, near the humans' 0.84. Pinned as
groundedness-v3.Why: Changing the judge or rubric is a measurement change; pinning means scores move only when the product does.
In real life
3 minChatbot Arena: humans vote
- Visitors compare two anonymous answers and vote
- Real human preference, at large scale
- Slow, and not about one company's tasks
The reference, too slow to gate our releases.
MT-Bench: a strong model judges
- Their published paper reported a strong judge agreeing with experts about as often as experts agreed with each other
- The same paper documented position, verbosity and self-preference biases
Fast and cheap, once agreement is measured and biases controlled.
What goes wrong
3 minNo criteria: scores cluster at 7–9 and drift with length and tone.
One criterion, anchored levels, evidence before verdict.
A judge that always says "grounded" scores high and catches nothing.
Kappa plus the confusion matrix.
Self-preference inflates scores for its own style.
A judge from another family, validated on human labels.
Trade-offs
3 min| Judging | Choose when | You pay |
|---|---|---|
| Pointwise | Monitoring a level, gating on thresholds | Scores cluster; leniency hides failures |
| Pairwise (both orders) | Choosing between two prompts or models | Two calls per pair; relative only |
| Cheap judge + escalation | Large sets, daily sampling | Two judges and a rule to calibrate |
- Karim · CTOThe strong judge costs ~0.004 EUR per case. The cheap one is ten times cheaper. Use it daily, strong only before releases.
- Lina · Senior engineer, your mentorThe cheap judge's kappa on our labels is 0.61. It would miss about twice as many bad answers.
- Karim · CTOThen what?
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- What the judge checks, one criterion per call: groundedness claim by claim, anchored 1–4, "plausible but absent = not found".
- The human truth it's measured against (150 labelled answers) and the bar: kappa near the humans' 0.84, and its catch rate on bad answers.
- How comparisons are judged (both orders) and which judge model: another family, pinned, re-checked on any change.
Our groundedness judge v1 says 97% grounded but caught only 7 of the 22 bad answers in our 150 human labels. Build judge v3 in `evals/judges/groundedness.ts`: list each claim, quote supporting source text or say "not found", then an anchored 1–4 score; plausible but absent counts as not found. Use another model family, pin its version, return structured output. Run v1 and v3 on the calibration set: confusion matrix, agreement, kappa, disagreements. Don't change the human labels. Judge pairwise comparisons in both orders.
- The confusion matrix: v3 catches ~19 of 22 bad answers (v1 caught 7).
- Kappa near the humans' 0.84; 89% agreement with kappa 0.40 isn't enough.
- Read 5 disagreements yourself: judge wrong, or rubric unclear?
- The judge prompt tuned until scores look good, without human labels.
- Human labels "fixed" to match the judge.
- A pairwise win rate from one order only.
Check yourself
2 minNext episode: three prompt regressions reached guests last month. Lina hands you the release process: make every change prove itself before it ships.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Why it matters, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
Lina and you calibrate the groundedness judge
Turn the v1 judge (passes 15 of 22 bad answers) into one we can trust to block a release.
- 1Get human labels
Two support leads label the same 150 answers (128 grounded, 22 not), independently. Compute their agreement: kappa 0.84.
Why: The labels are the reference, and the humans' own agreement is the ceiling. If they disagreed a lot, the definition of "grounded" would need fixing first.
- 2Read the confusion matrix
Run v1 on the 150. Agreement 89% looks fine, but the matrix shows 15 cases "human: not grounded, judge: grounded".
textjudge: pass judge: fail human: pass 126 2 human: fail 15 7 <- the cell that ships bugsWhy: Agreement is inflated when one label dominates. The "judge pass, human fail" cell is the regressions you'd ship.
- 3Compute kappa
Observed 0.887; expected by chance 0.853 × 0.94 + 0.147 × 0.06 ≈ 0.811. Kappa = (0.887 − 0.811) / (1 − 0.811) ≈ 0.40.
Why: Kappa removes the agreement a judge would get by always saying "grounded". 0.40 is weak.
- 4Fix the rubric
Read the 15 misses: all contain a plausible fee or time not in the sources. Rewrite: list claims, quote evidence for each, anchored 1–4 scale, "plausible but absent = NOT FOUND", ignore length. Switch to a judge from a different model family.
Why: Most disagreements live in an edge case the rubric didn't define. Claim-level checking turns a vibe into a verification.
- 5Re-measure and pin
v3: 19 of 22 caught, 4 false alarms, kappa ≈ 0.82. Pin the judge model and rubric
groundedness-v3, stored next to every score.Why: Changing the judge or rubric is a measurement change. Pinning means a score moves only when the product does.
| Bias | What it looks like | Mitigation |
|---|---|---|
| Position bias | Verdict flips when A and B swap places | Judge both orders; count only consistent wins |
| Verbosity bias | Longer answers score higher, even when padded | Rubric says length isn't a criterion; check score vs length |
| Self-preference | Judge favours its own model family's style | Use a judge from another family, or confirm on human labels |
| Leniency | Almost everything passes | Anchored low scores; claim-level checks; watch the judge-pass/human-fail cell |
| Drift | Same rubric, different numbers after a judge update | Pin the judge version; re-run calibration on any change |
Interactive simulation
Visual diagram
A judge pipeline: humans label a calibration set, the rubric is tuned until judge and humans agree, and only then the judge grades the full eval and production samples.
Analogy
- A new wine critic"Good wine, 1 to 10?" gets all 7sAn LLM judgeA vague rubric gives useless scores
- A new wine criticA sheet: acidity, balance, finishAn LLM judgeA clear rubric with described scores
- A new wine criticPreferring the first glass tastedAn LLM judgePosition bias, fixed by swapping order
- A new wine criticFavouring a former employer's styleAn LLM judgeSelf-preference bias
- A new wine criticBlind-tasting wines already scoredAn LLM judgeCalibrating against human labels
Where the analogy breaks: a human critic changes slowly. A judge model can change overnight when its provider updates it, so you re-check it whenever the judge, rubric or data changes.
Why it matters
Code checks can't grade the things that matter most to a hotel. Did the assistant invent a pet policy? Did it answer both halves of "can I bring my dog AND park?" Was the refusal polite?
People can grade these, but slowly: a minute or two per answer. Grading 500 answers for 20 prompt versions is about 170 hours. A judge does it in minutes, for a few euros.
The catch: a badly built judge gives confident, wrong numbers. The team ships a regression with a green dashboard. That's worse than no eval at all, because nobody is looking anymore.
Check your understanding
Practice ladder
Compute a judge's kappa
Estimation Warm-upA cheap judge was run on the 150-answer calibration set:
| judge: pass | judge: fail | |
|---|---|---|
| human: pass | 120 | 8 |
| human: fail | 10 | 12 |
Compute percent agreement and Cohen's kappa, and say how many bad answers it would let through on a 420-case golden set with the same proportions.
Anchor the helpfulness rubric
Design PracticeLina started a helpfulness rubric for concierge answers. Fill in the anchors for scores 3 and 2, and add one edge-case rule.
Criterion: does the ANSWER resolve the guest's QUESTION?
4 = Fully answers the question, with the next step if one is needed
(e.g. "Yes, late check-out until 13:00 is free; just tell reception.").
3 = ___
2 = ___
1 = Doesn't address the question, or answers a different one.
Edge cases:
- Length and friendliness are not criteria.
- ___A judge for policy compliance
Design PracticeMaya wants to track whether the concierge's answers respect StayHub's policies (no promising refunds outside the rules, no discounts, no sharing other guests' details). Design the judge: what code can check instead, the rubric outline, the calibration plan, and when it may gate releases.
Let the judge grade tone?
Decision StretchMaya also wants a "tone" score (warm, professional, on-brand) on every release. You labelled 100 answers with two people: human-human kappa 0.45. The best judge you tried reaches 0.41 against each human.
Write a short decision: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
A product manager says: "Just ask the model if the answer is good. Why do we need human labels at all?" Explain in your own words why, and how you'd make the judge trustworthy.