Skip to content

AI Evaluations

Testing software that gives a different answer every time.

correctnessgroundednessrelevanceinstruction followingtool-use accuracyhallucination rateLLM-as-judgeregression evals

Lessons

  1. 28.1Evals: testing software that never answers the same way twiceAI gives a different answer every time, so normal tests don't fit. Evals measure how often it's right, so you know if a change helped. 30 min Eval playground
  2. 28.2LLM-as-judge: grading with a model you don't fully trustYou can ask an AI to grade another AI's answers. It's fast and cheap, but only trustworthy once you've checked it against people. 30 min Eval playground
  3. 28.3Eval-driven development: shipping AI changes like a disciplined teamTurn every AI bug into a test case, check every change against all of them, and stop guessing whether your AI got better. 30 min Eval playground

On the job: try it on your own project

Pick one AI feature you own or plan to ship. Collect twenty real inputs with what a good answer looks like. Ask Claude Code to turn them into a small eval script on a branch, with simple checks plus an LLM-as-judge rubric for the rest. Run it on the current prompt. Write down the pass rate and the bar a change must meet to ship.

Can you do this now?

0/4. Anything you can't tick yet: reopen that lesson's practice ladder.