Module 10 of 11
Evaluation and model adaptation
- Production
- 4 lessons
- 1 h 45 min
Chapter 10 · The Lowtide Sleeper
Before dawn the Sleeper rests in the Repair Siding. The night has left a pile of real logbooks and complaint slips, and Marta wants evidence before anyone touches anything. You will turn the pile into a fair test, decide whether a failure calls for clearer orders, better pages or a retrained Voice, and choose the smallest change that fits tonight.
What you will be able to do
- Build and interpret evaluations of LLM systems.
- Decide between prompting, RAG and fine-tuning, and know how LoRA, QLoRA and DPO work.
Lessons
Evaluating LLM systemsCompleted
Read first: Observability and eval pipelines
Fine-tuning vs RAG vs promptingCompleted
Read first: RAG architecture and chunking, Prompting fundamentals
Fine-tuning methods: SFT, LoRA, QLoRA and DPOCompleted
Read first: Fine-tuning vs RAG vs prompting
Fine-tuning data, risks and shippingCompleted
Read first: Fine-tuning methods: SFT, LoRA, QLoRA and DPO
End of chapter 10 · The Lowtide Sleeper
Marta wrote it all down for once, and the thermos was, for the first time all night, still full.