Observability and eval pipelines

Story

Hide story

By two o’clock the announcements about the 02:40 connection have been wrong for an hour. Nobody on the crew noticed. The passengers did, one at a time, in the corridor.

Marta looks at the wall where a dashboard might hang. There are hooks and no dashboard. The departure board still reads ON TIME.

Where should the first warning have come from?

Concept#

Ordinary software is deterministic enough to unit-test: same input, same output, assert equality. An LLM system is not. Output varies from run to run, “correct” is often a judgement, and the behaviour depends on a chain of parts (prompt, model, retrieval, chunking, tools) that each change on their own schedule.

Production LLM systems therefore need two pieces of infrastructure that work together.

• Tracing and monitoring (online): record what happened on every real request, and watch aggregate health signals live.
• An evaluation pipeline (offline)Offline evaluationRunning a fixed test set against a change before release. It catches regressions, including rare intents.Open the glossary entryClose: run a labelled golden setGolden setA curated set of questions with known good answers or sources, used to score a system on every change. Keep slices tagged and add real failures over time.Open the glossary entryClose of cases against every proposed change before it ships, like a regression test suite for a system that cannot be asserted with ==.

Why It Matters#

Without them you cannot answer “did that prompt change make things better or worse?” You are left with anecdotes, and anecdotes miss regressions. A tweak that fixes the case you tried can quietly break ten others. A user complaint with no trace cannot be debugged, because you do not know what was retrieved, which prompt version ran, or what the raw response was.

Mental Model#

An aircraft has two safety systems. The flight recorder captures what actually happened on every flight, so that any incident can be reconstructed and the fleet’s health can be watched. The test rig runs every design change through known scenarios before a plane ever leaves the ground.

Traces are your flight recorder. The golden set is your test rig. The recorder tells you something went wrong after users met it. The rig tells you before. A mature team feeds the recorder’s worst flights back into the rig as new scenarios.

What to record on every request#

A traceTraceThe recorded path of one request through a system, with each stage's inputs, outputs, timing and cost. Traces hold user data and need privacy controls.Open the glossary entryClose is a tree of steps (spans) under one trace id. For a RAG request that means rewrite, retrieve, rerank, build prompt and generate, each with its own timing.

OpenTelemetry maintains semantic conventions for GenAI spans, metrics and events, so check them (and their current status) before inventing your own attribute names (repository, accessed 2026-10-09).

Privacy: traces are sensitive data#

A trace holds user text, retrieved company documents and model output. Treat the trace store as production data.

• Redact personal data and secrets before storing, or store identifiers and hashes instead of text where you can.
• Limit who can read full prompts; give most engineers redacted views.
• Set a retention period and delete on schedule. Deletion requests (for example under privacy law; check your own legal requirements) must reach traces, caches and exported datasets.
• Sample full-content capture if you do not need every request.
• Do not copy production traces into a golden set without the same redaction.

Online monitoring and alerts#

Online signals are fast, real and noisy. Alert on change relative to your own baseline, not on numbers copied from a blog.

• Reliability: error rate, timeouts, retry and fallback rate.
• Latency: p50, p95 and p99 per stage; time to first token.
• Cost: cost per request and per tenant; input token count (prompt growth is a classic silent cost leak).
• Behaviour: abstention (“I don’t know”) rate, empty-retrieval rate, guardrail trigger rate, refusal rate. A sudden jump either way means something changed.
• Quality proxies: thumbs-down and regeneration rate, and the score of a sampled groundedness or judge check.
• Drift: shifts in query topics, retrieval score distribution or document freshness.

Limits: few users click feedback, those who do are not representative, and a metric moving tells you something changed but not what to fix. Alerts cover the failures you thought of.

Offline evaluation and the gate#

A golden set is a collection of inputs with expectations about the output. For a RAG system build it from several kinds of case: should retrieve and answer, should say “I don’t know”, should ask a clarifying question, adversarial or injection attempts, and every past incident. Tag each case with a slice (topic, tenant type, language, difficulty).

Run it on every change to the prompt, model, retriever, chunker or embedding model, and compare with the current baseline. Retrieval can be scored separately (did the right chunk id appear in the top k?) from generation (was the answer faithful and complete?), which is why the trace stores retrieved ids (RAG evaluation and hallucination, Evaluating LLM systems). If a model acts as the judge, calibrate it against human labels first.

Read results per slice and with noise in mind. An average hides collapses, and a small set is noisy (the difference between two runs is noisier still).

Worked example (illustrative numbers)#

A golden set has 100 cases: 80 general, 10 refund questions, 10 date questions. Prompt v11 passes 72, 9 and 9, which is 90 in total. Prompt v12 passes 78, 5 and 8, which is 91 in total.

• The headline moved from 90 percent to 91 percent. The sampling noise on a pass rate near 0.9 with 100 cases is about sqrt(0.9 x 0.1 / 100) = 0.03, three points, so a one-point gain is within noise.
• The refund slice fell from 9 of 10 (90 percent) to 5 of 10 (50 percent). With only 10 cases this is also noisy, but it is a large drop, and it is exactly the kind of thing a single average hides. The right response is to read the failing refund cases, not to ship.

Closing the loop#

1. Sample production traces, weighted toward thumbs-down, abstentions, low-confidence retrievals, escalations and new topics.
2. Label them (people, or a calibrated judge checked by people). Fix the prompt, the retrieval or the data.
3. Add the labelled cases to the golden set with their slice tags, so that failure is caught if it returns.
4. Gate changes on the set (quality, latency and cost thresholds), then roll out in stages (RAG security, reliability and release).

Diagram#

Diagram: Every request is traced.

Real-World Application#

A team ships a prompt change that shortens answers and saves tokens. The offline gate passes on average, but a per-slice read shows the refund slice dropped. They inspect the failing cases, restore the missing instruction, and add three new refund cases from production traces. After release, the dashboard shows the abstention rate stable and cost per request down.

Interview Perspective#

Expect: “how do you know if a prompt change made things better or worse?” and “what do you log for every LLM request?”. A strong answer names the trace fields, separates online monitoring from the offline gate, mentions slices and noise, and raises privacy unprompted. Weak answers stop at “manual testing” or “user feedback”.

Provenance: Common industry pattern. These are standard GenAI production follow-ups, not a verified list for any named company.

Practice Questions#

• Basic: What should be logged for every LLM request in production?
• Intermediate: Why is a golden eval set needed even when you have good user feedback signals?
• Advanced: How would you design an eval set that catches retrieval regressions and generation regressions separately?
• Product-Company: A prompt change reduces average response length (cheaper) but some users say answers feel incomplete. How do you investigate?

Activity#

List the exact fields you would log for one RAG request, from the user question to the final answer. For each field, write the question it lets you answer, and mark which fields need redaction.

Homework#

Design a 15-case golden set (questions plus expected-answer criteria) for a system of your choice, split across “should retrieve and answer”, “should say I don’t know” and “should ask a clarifying question”. Tag every case with a slice. Then write the three alerts you would set first and the baseline each one is compared with.

Quick Revision#

• LLM systems need tracing (flight recorder) and an offline eval gate (test rig) because output is variable and the pipeline has many moving parts.
• Trace versions, retrieved ids and scores, tokens, latency, cost and outcome signals, per request.
• Online monitoring is live but noisy. Offline eval catches regressions before users do.
• Read results per slice, and remember small sets are noisy.
• Traces hold sensitive data: redact, restrict, expire.
• Feed bad production cases back into the golden set.

Check yourself#

• Walk-through: Specify what you log per LLM request, what you alert on, and how production traces feed back into the eval set.
• Variation: Prompt version 12 regressed a rare intent. How would your pipeline have caught it before deploy?
• Common misconception: Logging only final answers; no prompt or version tracking; no privacy policy; trusting one average score.

Related#

• RAG evaluation and hallucination
• Cost and latency optimization
• Production deployment of RAG
• RAG security, reliability and release