Lesson deck
RAG evaluation in depth flashcards
Reviewing needs JavaScript. You can read every card in the list below.
Loading your cards...
Back to the lesson6 cards in this deck
Read all 6 cards as a list
- Front
- What's the difference between a retrieval metric and a generation metric in RAG eval?
- Back
- Retrieval metrics (precision@k, recall@k, MRR) measure whether the right chunks were found; generation metrics (faithfulness, relevance) measure whether the answer is grounded in and addresses those chunks.
- Front
- Define recall@k, precision@k and MRR, and how they differ.
- Back
- Recall@k: was the relevant item found in the top k. Precision@k: fraction of the top k that is relevant. MRR: mean of 1/rank of the first relevant result, 0 if none; it also rewards ranking it higher.
- Front
- Three ranks 3, 1, none: what is MRR?
- Back
- (1/3 + 1 + 0) / 3, about 0.44. A miss counts as 0 and stays in the denominator.
- Front
- Retrieved gold = no, faithful = no, correct = yes. What does that mean?
- Back
- A lucky answer from the model's memory, not the corpus. It hides a retrieval gap and cannot be cited.
- Front
- Name three LLM-judge pitfalls and mitigations.
- Back
- Position bias, verbosity bias, self-preference. Use a fixed rubric, swap order in pairwise comparisons, calibrate against human labels, and prefer a judge from another model family.
- Front
- Why can synthetic evaluation questions mislead?
- Back
- They tend to copy chunk wording and be easier than real queries, inflating scores. Review a sample and add real logged queries and unanswerable questions.