Lesson deck
Evaluating LLM systems flashcards
Reviewing needs JavaScript. You can read every card in the list below.
Loading your cards...
Back to the lesson6 cards in this deck
Read all 6 cards as a list
- Front
- Why can't a public benchmark pick your model?
- Back
- Contamination, saturation and domain mismatch. Use benchmarks to shortlist; decide on your product eval.
- Front
- How do you validate an LLM judge before trusting it?
- Back
- Label a sample by humans, compute Cohen's kappa against the judge (not raw percent agreement), report the label balance, and re-check periodically.
- Front
- Why can high percent agreement between judge and humans mislead?
- Back
- When one label dominates, guessing it already agrees often. With 95% passes, chance agreement is about 0.905, so 90% observed is no better than chance. Use kappa and look at the rare class.
- Front
- What is the difference between a dev set and a held-out set?
- Back
- Iterate on the dev set. Use the held-out set only to gate releases. Tuning against it turns it into a second dev set and its score stops meaning anything.
- Front
- What should you do before choosing evaluation metrics?
- Back
- Error analysis: read 50 to 100 real failing traces, sort them into failure categories, and measure the top categories.
- Front
- Why is a +3 point gain on 100 cases weak evidence?
- Back
- Sampling noise: at a pass rate near 0.8, one score is uncertain by roughly 8 points either way (illustrative). Compare variants paired on the same cases, with a bootstrap interval, or add cases.