Intermediate project
RAG with an eval harness
Builds on 3 lessonsabout 1 h 30 min of reading first
In this project
What you need first
- Completed:RAG architecture and chunkingRetrieval-augmented generation
- Completed:Better retrievalRetrieval-augmented generation
- Completed:RAG evaluation in depthRetrieval-augmented generation
Skills you will practise
- Document processing
- Retrieval
- Retrieval-augmented generation
- RAG evaluation
Problem
Build a RAG system you can measure, and use the measurements to decide what to change.
Requirements
- A corpus of 50–200 documents.
- Hybrid retrieval plus a reranker.
- Citations and an abstain path.
- A golden set of at least 40 queries, including unanswerable ones.
- A report of recall@k, MRR and faithfulness.
Prerequisites
Tasks
- To do: Assemble the corpus and ingest it.
- To do: Implement hybrid retrieval and a reranker.
- To do: Add citations and an abstain path.
- To do: Write the golden set (at least 40 queries).
- To do: Run the metrics and write the report.
- To do: Ablate chunk size and reranking, and record the effect of each.
Evaluation
Record real measurements and never invent numbers. The finished project should give you: retrieval-vs-generation diagnosis, a trade-off table and a regression gate.