Generative AI
SearchStart here

RAG evaluation in depth

  • Intermediate
  • 30 min
  • Lesson 6 of 6 in Retrieval-augmented generation
  • Last verified
In this module
On this page

Before you start

You will be able to

  • Design an evaluation that scores retrieval and generation separately, then end to end.
  • Compute recall@k, precision@k and MRR from ranked results by hand.
  • Turn misses into a failure taxonomy that says which stage is responsible.
  • Build a golden set that includes unanswerable questions, and use an LLM judge while knowing its pitfalls.
  • Run controlled experiments: one change at a time, on the same set, tracking quality, latency and cost together.

Story

Marta Voss leans on the counter, holding an empty thermos. “Give me one number. Is the Records Car fine, yes or no?”

Merrow lays out three complaint slips. One is a wrong page. One is an old page. One is the right page, read badly.

“One number can’t tell these apart,” he says. Which numbers could?

Concept

RAG evaluation is scoring the system with labelled questions so you can tell what is working, what is not, and which stage to fix. The principle: evaluate each stage separately, then end to end. A single “answer quality” score cannot tell you what to fix.

Why it matters

Without evaluation, every change is a guess. A new chunk size, a new embedder, or a new prompt can help one kind of question and quietly break another. The score you trust decides what you ship, so its limits matter as much as its value.

Mental model

Think of a restaurant inspection that checks the kitchen (retrieval), the plating (generation), and the whole meal (end to end). A single “customer satisfaction” number would not tell you whether the problem was the ingredients or the cooking.

Golden set
question, gold answer, gold chunks

Retrieval eval
recall@k, MRR, nDCG

Generation eval
faithfulness, relevance, correctness

End-to-end eval

CI gate / dashboard

Production traces + feedback

Diagram summary: A golden setGolden setA curated set of questions with known good answers or sources, used to score a system on every change. Keep slices tagged and add real failures over time.Open the glossary entry feeds a retrieval evaluation and a generation evaluation, whose results combine into an end-to-end evaluation that gates releases in CI or a dashboard. Production traces and feedback flow back into the golden set.

Retrieval metrics

These need labelled relevant chunks (or passages) for each query.

  • Hit rate or recall@kRecall@kThe share of the relevant or true top-k items that appear in the top k results. For an ANN index it measures the recall lost to approximation.Open the glossary entry: the fraction of queries (or of relevant chunks) found in the top k. The primary retriever metric: if the evidence is not retrieved, generation cannot fix it.
  • Precision@k: the fraction of the top k that is relevant. Low precision means a noisy prompt.
  • MRR (mean reciprocal rankMean reciprocal rank (MRR)The average of 1/rank of the first relevant result across queries, counting 0 for a miss. It rewards ranking the right item higher.Open the glossary entry): the average of 1 / rank of the first relevant result, counting 0 when none is found. Good when one good chunk suffices.
  • nDCGnDCGNormalized discounted cumulative gain: a ranking metric that rewards putting highly relevant items earlier and handles graded relevance, not just relevant or not.Open the glossary entry@k: rewards putting highly relevant items earlier and handles graded relevance.
  • Context precision and recall (RAGAS-style): whether relevant context ranks high, and whether the retrieved context covers what the gold answer needs (Es et al., 2023).

Worked example

Three queries, top 5 results. Query A: relevant chunk at rank 2. Query B: rank 1. Query C: not in the top 5.

  • Recall@5 = 2/3, about 0.67 (two of three queries found it).
  • MRR = (1/2 + 1 + 0) / 3 = 0.5.

Precision@5 for a single query whose top 5 holds relevant chunks at ranks 1, 3 and 4 is 3/5 = 0.6: relevant results in the top k, divided by k.

A second MRR set: ranks 3, 1, none. MRR = (1/3 + 1 + 0) / 3, about 0.44.

Note what each tells you. Recall@5 says whether the chunk was found. MRR also says how high it ranked, which matters if you only pass the top few to the prompt.

Generation metrics

  • FaithfulnessFaithfulnessWhether an answer's claims are supported by the retrieved context. It is a generation metric, separate from whether the answer is correct.Open the glossary entry or groundedness: is every claim supported by the retrieved context? Typically: split the answer into claims, check each against the context with an LLM judge or an entailment model.
  • Answer relevance: does it address the question?
  • Answer correctness: does it match the gold answer (semantic match or judge)?
  • Citation accuracy: do cited sources support the cited claims?
  • Abstention quality: does it say “I don’t know” when the context lacks the answer? Test with unanswerable questions.

Failure taxonomy

Combine the signals to decide who is responsible for each miss.

Retrieved gold?Answer faithful?Answer correct?Diagnosis
No-NoRetrieval failure (fix chunking, embeddings, hybrid, filters)
YesYesNoSource wrong or ambiguous, or the question is ambiguous
YesNoNoGeneration failure (prompt, model, noisy context)
NoNoYes (by luck)Answer came from the model’s memory, not the corpus: risky
YesYesYesHealthy

A quick manual check agrees with the table: paste the gold passage into the prompt. Correct afterwards means the retriever was the problem.

Building the golden set

  • Start with real queries from logs and subject-matter experts. Tens of questions are enough to begin, and a few hundred are better. Cover types: factual, multi-hop, ambiguous, unanswerable, exact-ID and long-form.
  • Label gold chunks or documents, not just final answers, so retrieval can be scored.
  • Synthetic questions (an LLM writes questions from chunks) scale the set. Review a sample by hand: synthetic questions tend to be easier and copy the chunk’s wording, which inflates scores.
  • Keep a frozen regression set and a rotating fresh set from production, so you do not overfit.
  • Slice results by query type, document type, tenant and language. An average can hide a failing slice.

LLM-as-judge

A judge model scores answers at scale. It has known pitfalls: position biasPosition biasA judge's tendency to favour an answer because of where it appears in the prompt. Swapping the order in pairwise comparisons helps.Open the glossary entry, verbosity bias, self-preference (favouring answers from its own model family) (Zheng et al., 2023). Judges can also give different scores on repeated runs. Mitigate with a fixed rubric, structured output, a low temperature where the model allows it, repeated judging or a majority vote (temperature 0 alone does not make a judge deterministic, and some current models restrict non-default sampling parameters; see the Anthropic Claude Sonnet 5.5 migration guide, checked 2026-10-09 in resources.md), pairwise comparisons with the order swapped, calibration against human labels (report agreement), and a judge from a different model family when possible.

Online and production signals

Thumbs up or down, copy and edit rate, follow-up rephrasing, escalation to a human, “no answer” rate, citation clicks, latency and cost per query. Sample low-signal traces for human review and add failures to the golden set.

Experiment discipline

  • Change one thing at a time (chunk size, embedder, reranker, prompt) and compare on the same set.
  • Report uncertainty. Small sets are noisy; one question on a 40-question set is 2.5 points.
  • Track quality, latency and cost together. A gain of two points of recall that doubles latency may not ship.
  • Gate deploys on thresholds. See RAG security, reliability and release.

Tools

Examples include RAGAS, TruLens, DeepEval, LangSmith and Arize Phoenix, plus plain scripts for recall@k and MRR. Features change often, so check current documentation before choosing.

Real-world application

Teams wire the golden set into CI so a change that drops recall@k on a slice fails the build, and they review a sample of production traces weekly. The RAG evaluation harness project (listed under Projects) builds this end to end.

Interview perspective

Provenance: common industry pattern; no company-specific claim.

Interviewers probe whether you evaluate stages separately, can compute recall@k and MRR, know that LLM judges are biased, and treat small score differences with suspicion. Name a metric for each stage and the action each one would trigger.

Practice questions

Provenance: generated practice.

  • Why is recall@k the first metric to look at?
  • Compute MRR for ranks 3, 1, none.
  • How do you test abstention?
  • Why can synthetic evaluation sets mislead?

Activity

Do, notice, apply.

  1. Do. Invent five queries with ranked results (rank of the first relevant chunk, or none). Compute recall@5, precision@5 if you invent the other relevant flags, and MRR by hand.
  2. Notice. Change one query’s rank from 2 to 1 and from none to 5. Which metric moves more, and why?
  3. Apply. For each miss, write the failure-taxonomy row and the first fix you would try.

Homework

Take the 10-question golden set from the previous lesson and extend it with gold chunk ids for each question and at least two unanswerable questions. Write the rubric a judge model would use for faithfulness.

Quick revision

  • Evaluate stages separately, then end to end.
  • Recall@k: found it? MRR: how high? Precision@k: how noisy?
  • Faithfulness, relevance, correctness, citation accuracy, abstention.
  • The taxonomy table names who is responsible for each miss.
  • Judges are biased: calibrate against human labels.
  • One change at a time, report uncertainty, track quality, latency and cost.

Check yourself

  • Checkpoint (answer aloud, no notes): Design an evaluation that separates retrieval from generation for a new RAG assistant.
  • Re-test (nearby variant): Compute MRR and recall@3 for ranks 2, 5, 1, none.
  • Misconception to avoid: one “answer quality” score is enough; synthetic questions are as hard as real ones.

Practice

Test what you just read. Quizzes and cards are grouped by module: Retrieval-augmented generation.

Check yourself12 questions on what this lesson set out to teach

Checking answers needs JavaScript. You can still read every question and option below.

0 of 12 answered

Keyboard: arrow keys move between options. Space or Enter selects the focused option.

ObjectiveDesign an evaluation that scores retrieval and generation separately, then end to end.

  1. Question 1 of 12Why does a single 'answer quality' score make a poor RAG evaluation?

    Choose one answer.

  2. Question 2 of 12Marta asks for one number to say the Records Car is fine. After a change, end-to-end accuracy is unchanged at 80%, but recall@5 fell and faithfulness rose. What is the most useful reading?

    Choose one answer.

ObjectiveCompute recall@k, precision@k and MRR from ranked results by hand.

  1. Question 3 of 12Three queries return the first relevant chunk at ranks 4, 2 and none (top 5). What are recall@5 and MRR?

    Choose one answer.

  2. Question 4 of 12Compute MRR for four queries whose first relevant chunk is at ranks 4, 1, 2 and none.

    Choose one answer.

  3. Question 5 of 12For one query the top 5 results are ranked 1 to 5, and the relevant ones are at ranks 2 and 4. What is precision@5?

    Choose one answer.

ObjectiveTurn misses into a failure taxonomy that says which stage is responsible.

  1. Question 6 of 12In a failure analysis, gold chunks were retrieved, the answer is faithful to them, yet the answer is wrong. What does the taxonomy say?

    Choose one answer.

  2. Question 7 of 12A query's gold chunk was not retrieved, the answer is not faithful to the retrieved context, but the answer happens to be correct. Why is this still a concern?

    Choose one answer.

ObjectiveBuild a golden set that includes unanswerable questions, and use an LLM judge while knowing its pitfalls.

  1. Question 8 of 12Which golden-set design is strongest for testing a corpus-backed assistant?

    Choose one answer.

  2. Question 9 of 12A team uses the same model family to generate answers and to judge them, and the judge always prefers the longest answer. Which two pitfalls does this suggest?

    Choose all that apply.

  3. Question 12 of 12Which statement about frozen and fresh evaluation sets is correct?

    Choose one answer.

ObjectiveRun controlled experiments: one change at a time, on the same set, tracking quality, latency and cost together.

  1. Question 10 of 12A team tests a new embedder, a new chunk size and a new prompt in a single release. On a 40-question set, recall@5 rises 2.5 points (one question) while latency doubles. What is the main experiment-design problem?

    Choose one answer.

  2. Question 11 of 12A reranker lifts recall@5 from 0.80 to 0.83 on a large golden set but adds a noticeable delay to every query. What is the right way to decide whether to ship it?

    Choose one answer.

Build with this

On this page