Evaluating LLM systems

Story

Hide story

Before dawn the Sleeper rests in the Repair Siding. Marta has dragged a crate of last week’s logbooks into the lamp-light: every announcement Tobias made, and what passengers said afterwards.

“That’s the test,” she says, and shakes her thermos. Empty. Merrow offers to pull out the ones that went well, to keep things tidy. Marta looks at him for a while.

What would a fair test look like?

Concept#

Evaluating an LLM system means measuring, with repeatable checks, whether the system does its job and whether a change made it better or worse. Outputs are open-ended and can differ between runs, so you cannot write one assertion and be done. Evaluation becomes a small system of its own: datasets, graders, metrics, thresholds and a feedback loop from production.

This lesson is the general method. RAG evaluation in depth covers retrieval metrics, golden sets and the RAG failure taxonomy, and Observability and eval pipelines covers tracing and running evals in production. Here the focus is the layers, the judge, and the decision.

Why it matters#

Every change to a prompt, model, retriever or tool can fix one kind of input and quietly break another. Without a repeatable eval, “it looks better” is the only evidence, and you find regressions through user complaints. “How do you know it is good, and that this change did not break something?” is a standard question in GenAI design and project rounds.

Mental model#

Think of a test pyramid. At the bottom sit many cheap, deterministic checks that run on every change. In the middle sit fewer, costlier model-graded checks. At the top sit a thin layer of human review and live experiments, which are the most trustworthy and the slowest.

Diagram: The pyramid stacks five layers from cheap deterministic checks up to online signals.

Start with error analysis#

Before choosing metrics, read a sample of real failing outputs, often 50 to 100 traces, and sort them into failure categories. The top categories tell you what to measure. Metrics chosen without this step tend to measure what is easy rather than what is breaking, and the score improves while users still complain.

The layers#

1. Deterministic checksDeterministic checkA rule-based test such as schema validity, a regex or unit tests. It is fast, free and consistent.Open the glossary entryClose. Schema validity, regex or substring matches, tool-call arguments, a citation being present, length limits, forbidden content, unit tests for generated code. They are fast, free, and never disagree with themselves.
2. Reference-based metrics. Exact match or F1 for extraction tasks, where a gold answer exists. ROUGE and BLEU compare word overlap with a reference and are weak for open-ended text, since many different answers are good. Embedding similarity is a coarse signal.
3. LLM-as-judgeLLM-as-judgeUsing a model to score outputs against a rubric or pick the better of two. It scales, but has its own biases and must be validated against human labels.Open the glossary entryClose. A model scores an output against a rubric, or picks the better of two outputs. It scales to qualities that code cannot check, such as helpfulness or tone, at the price of its own errors (next section).
4. Human review. Experts label hard cases, subjective quality and safety, and they calibrate the judge. It is the slowest and most expensive layer, so you use it where it matters.
5. Online signals. A/B tests, edit and retry rates, abandonment, thumbs and escalations to a human. They measure real use, but they are noisy and arrive after release.

Push as many checks as you can down the pyramid. A check that code can do should not be paid for in judge calls.

LLM-as-judge: use it, but validate it#

A judge model scores at scale, and Zheng et al. (2023) found that strong judges can agree with human preferences more than 80% of the time on their chat benchmarks, about as often as two humans agree with each other. The same work documents biases: position bias (favouring an answer because of where it appears), verbosity bias (favouring longer answers), self-enhancement bias (favouring outputs from its own model family) and limited reasoning on hard items such as maths.

Mitigations:

• A clear rubric with anchored levels, and structured output.
• Pairwise comparison, run in both orders.
• A reference answer for questions that have one.
• A low temperature where the model allows it. Temperature 0 alone does not make a judge deterministic, and some current models restrict non-default sampling settings, so repeat the judging or take a majority vote on important decisions (see RAG evaluation in depth and the migration guide listed above).
• A judge from a different model family than the system being judged.
• Measure agreement with humans on a labelled sample before you trust it, and re-check periodically because prompts, models and data drift.

Worked example: agreement beyond chance#

Raw agreement can flatter a judge. Suppose 100 outputs are labelled pass or fail by a human and by the judge. They agree on 85, so observed agreement is 0.85. If both label about half the items pass and half fail, chance alone would produce agreement of about 0.5 (0.5 × 0.5 for both-pass plus 0.5 × 0.5 for both-fail).

Cohen’s kappaCohen's kappaA measure of agreement between two raters, such as judge and human, corrected for agreement expected by chance. Raw percent agreement misleads when one label dominates.Open the glossary entryClose removes that chance part: kappa = (observed − chance) / (1 − chance) = (0.85 − 0.5) / (1 − 0.5) = 0.70. If the labels were 90% pass, chance agreement would be 0.9 × 0.9 + 0.1 × 0.1 = 0.82, and the same 85% observed agreement would give kappa = (0.85 − 0.82) / 0.18, about 0.17. The judge looks accurate and adds little, because it can score well by guessing the common label. Report kappa and the label balance, not just percent agreement.

Datasets#

• Draw cases from production logs and from experts. Include edge cases, adversarial inputs, unanswerable questions, multiple languages, long inputs and cases where the right behaviour is to refuse.
• Record why an answer is right, not just what it is, so reviewers can audit labels.
• Version the dataset. Keep a dev set you iterate on and a held-out setHeld-out setData kept aside to gate releases. If you tune against it, it becomes another dev set and its score stops meaning anything.Open the glossary entryClose that only gates releases. Tuning prompts against the held-out set turns it into a second dev set, and the release number stops meaning anything.
• Synthetic cases scale coverage, but they skew easy and echo the generator’s phrasing. Hand-check a sample.
• Slice results by intent, language, customer or length. An average can stay flat while one slice collapses.

Is the difference real?#

A test set is a sample, so scores carry noise. With 100 cases and a true pass rate of 0.8, the standard error of the measured rate is sqrt(0.8 × 0.2 / 100) = 0.04, and an approximate 95% interval is ±1.96 × 0.04, about ±8 points. The numbers are illustrative, but the effect is real: a +3 point change on a set that size is well inside the noise.

Practical habits:

• Compare variants on the same cases and look at the paired differences, which cancels case difficulty. A bootstrapBootstrapResampling the test cases with replacement many times and recomputing a metric each time, to get an uncertainty interval without a formula.Open the glossary entryClose (resample the cases many times, recompute the difference) gives an interval without a formula.
• If the interval includes zero, you have not shown an improvement. Add cases or accept that the change is neutral.
• Report quality together with safety, latency and cost. A gain that doubles latency may not ship.
• Roll out in stages: shadow traffic, then a canary, then full release, watching the online signals.

Benchmarks versus product evals#

Public benchmarks help shortlist models, but they have three weaknesses. ContaminationBenchmark contaminationBenchmark content having leaked into a model's training data, which inflates its scores. Use public benchmarks to shortlist, not to decide.Open the glossary entryClose: test items leaked into training data. Saturation: top models cluster near the ceiling, so differences vanish. Domain mismatch: the benchmark’s tasks are not your tasks, language, format or risk profile. Use a benchmark to decide which models to try, and your product eval to decide which one ships. When they disagree, check the product eval’s sample and labels first, then trust it, because it measures your workload.

Real-world application#

For example, a support assistant’s CI might run 400 cases on every prompt change. Deterministic checks cover format and citations, a judge scores helpfulness (validated against humans on a labelled sample, for example kappa around 0.7; illustrative), and a held-out set gates the release. A weekly review of sampled production traces feeds new failures back into the dev set.

Interview perspective#

Provenance: common industry pattern; no company-specific claim.

Expect “how would you evaluate a summariser, chatbot or agent”, “can you trust LLM-as-judge” and “how do you detect regressions”. Strong answers name the layers, validate the judge against humans, and talk about noise and slices. Weak answers rely on a single average, trust the judge unvalidated, or tune on the held-out set.

Practice questions#

Provenance: generated practice.

• Basic: Name three biases of LLM judges.
• Intermediate: How do you validate a judge, and why is kappa better than percent agreement?
• Advanced: A public benchmark ranks model A above B, but your eval says B is better. How do you resolve it?
• Product: Design the eval for an email-drafting assistant used by 100,000 people.

Activity#

Do, notice, apply.

1. Do. For a summarisation feature, write five deterministic checks, a four-level judge rubric with an anchor example per level, and three slices to report.
2. Notice. Which of your five checks would a judge catch less reliably than code does?
3. Apply. Say which layer you would run on every commit, which nightly, and which before release.

Homework#

Build a 30-case eval set with expected criteria. Run two prompt variants over it, compute the difference with a simple bootstrap interval, and write a ship or no-ship note that mentions the noise.

Quick revision#

• Pyramid: deterministic, reference, judge, human, online. Push checks down.
• Error analysis first, then datasets, then gates.
• Judges are biased: validate with kappa, swap orders, use a rubric.
• Dev set to iterate, held-out set to gate; slice the metrics.
• Benchmarks shortlist, product evals decide.

Check yourself#

• Checkpoint (answer aloud, no notes): Your new prompt scores +3 points on the judge over 100 cases. Do you ship?
• Re-test (nearby variant): Design the eval for a legal-clause extractor where mistakes are costly.
• Misconception to avoid: the LLM judge is ground truth.

Related#

• RAG evaluation and hallucination
• RAG evaluation in depth
• Observability and eval pipelines
• Agent evaluation: state, trajectory, pass^k and release gates