Lesson deck
Observability and eval pipelines flashcards
Reviewing needs JavaScript. You can read every card in the list below.
Loading your cards...
Back to the lesson7 cards in this deck
Read all 7 cards as a list
- Front
- Why do LLM systems need an offline eval pipeline?
- Back
- Output varies and cannot be asserted with equality, and many parts change independently. A golden set run on every change catches regressions before users meet them.
- Front
- What do you record for each production LLM request?
- Back
- Trace id, prompt and model versions, inputs and outputs (privacy-safe), retrieved ids and scores, tokens, latency per stage, cost, outcome signals and errors.
- Front
- How do production failures feed back into the eval set?
- Back
- Sample traces (weighted toward thumbs-down, abstentions and escalations), label them, add them to the golden set with a slice tag after redaction, then gate future changes on the set.
- Front
- Why read golden-set results per slice?
- Back
- An average can rise while one slice collapses. Small sets are also noisy: with 100 cases near 90 percent the sampling noise is about 3 points, so one point is not a signal.
- Front
- What does online monitoring catch that offline eval misses, and the reverse?
- Back
- Online catches live outages, cost and latency shifts and drift in real traffic. Offline catches quality regressions, including rare intents, before release. Neither replaces the other.
- Front
- What privacy rules apply to LLM traces?
- Back
- They hold user text and company documents. Redact personal data and secrets, restrict access to full prompts, set retention and deletion, and apply the same rules before reusing traces in an eval set.
- Front
- Which alerts help for an LLM service and what are they compared with?
- Back
- Error and timeout rate, p95 latency per stage, cost per request, input token growth, abstention and empty-retrieval rate, feedback and sampled quality scores. Compare each with your own baseline, especially after deploys.