RAG security, reliability and release
- Production
- 25 min
- Lesson 5 of 5 in Production: cost, serving and observability
Select text in the lesson first to keep it with your note. With the keyboard, move focus here and press the button: the note goes with the section you are reading.
In this module
On this page
Before you start
- Completed:Production deployment of RAG
You will be able to
- Enforce permissions at retrieval, scope caches by tenant and document version, and choose what to degrade first when a dependency fails.
- Plan a release with an offline eval gate, shadow traffic, a canary and a rollback, versioning every component that can change.
Numbers below are illustrative. State your assumptions in interviews and never present them as facts.
Concept
Production deployment of RAG covered how a deployed RAG system is shaped, fed, sized and paid for. This lesson covers what keeps it safe and changeable once real users depend on it: who may see which document, what happens when a dependency fails, how you watch it, and how you ship a change without gambling. If you have not read the first lesson, the main things you need from it are the two-path picture (a fast read path that answers users, and a write path (ingestion) that keeps the index correct) and the index alias swap, which is how a release is undone.
Why It Matters
Security and release mistakes are the expensive ones. A missing permission filter or a shared cache key shows one customer another customer’s data. A dependency outage with no plan turns a slow reranker into a full outage. An untested chunker change quietly breaks one document type for weeks. Interviewers probe these because a demo never meets them. For each box on the whiteboard, say what it protects, what happens when it fails and how you would undo a change to it.
Mental Model
Think of three questions you ask about every part of the system.
- Who may see this? Permissions travel with the data and are checked where the data is fetched, not in a polite instruction to the model.
- What if it breaks? Decide before the incident which part degrades first, and keep the dangerous shortcuts (no permission filter, no retrieval) off the list.
- How do I undo it? Every change ships with a version, a gate and a way back.
Security and privacy
- Access control at retrieval: store permissions as metadata, filter by user or group at query time, and verify on the final context too. Never rely on the prompt to hide data.
- Indirect prompt injection: retrieved text is untrusted. Delimit it, tell the model to treat it as data, restrict tool permissions, filter outputs and vet ingestion sources (Guardrails, safety and prompt injection).
- PII and secrets: redact at ingestion or mask at output, encrypt in transit and at rest, and set retention. Deletion requests (check your legal requirements) must reach vectors, caches and logs.
- Audit and tests: log who retrieved what, and test tenant isolation.
- Vendors: check data residency and terms for hosted LLMs and embedders.
Reliability and graceful degradation
| Failure | Mitigation |
|---|---|
| LLM provider outage or rate limit | Retries with backoff, circuit breaker, fallback model or provider |
| Reranker down | Skip rerank and serve hybrid results (degraded but available) |
| Vector DB slow | Timeouts, replica failover, cached results |
| Empty or low-confidence retrieval | Abstain, ask a clarifying question, or escalate |
| Runaway agent loop | Max steps, token and time budgets |
| Stale index | Freshness monitor, alert on ingestion lag |
Decide in advance what degrades first. Dropping the reranker lowers quality slightly. Answering without retrieval risks hallucination, so abstaining is usually better.
Observability
Trace each request end to end and watch per-stage dashboards, as described in Observability and eval pipelines: query and rewritten query, retrieved ids and scores, reranked order, prompt token count, model, latency per stage, cost, guardrail outcomes, answer and feedback. RAG-specific dashboards: empty-retrieval rate, abstention rate, groundedness samples, cache hit rate, ingestion lag and recall on the golden set.
Release engineering
Version everything that can change: prompts, embedding model, chunker, index, reranker and LLM. Put changes behind feature flags and treat prompt edits like code. The 5 percent canary is an illustrative size.
Failure patterns to know
These are recurring patterns, not reports of specific incidents.
- Silent staleness: ingestion fails and answers drift out of date. Fix: freshness target plus an alert on ingestion lag.
- Embedding model swapped without re-indexing: queries and documents no longer share a space, so recall collapses. Fix: version the model and use the alias swap.
- Access-control bug: a restricted document is retrieved for the wrong user. Fix: filter at retrieval, test it, audit.
- Cross-tenant cache hit: a semantic cache returns another tenant’s answer. Fix: put tenant, permissions and document version in the cache key.
- Chunker change hurts tables: accuracy on one document type falls unnoticed. Fix: a regression set that gates the change.
- Cost spike: agent loops or growing context. Fix: step and token budgets and per-tenant cost alerts.
Interview whiteboard script (45 minutes)
Requirements and scale, then data sources and freshness, ingestion, chunking and embedding, index choice and sizing, retrieval (hybrid, rerank, filters), generation and prompting, guardrails and security, latency and cost budget, evaluation plan, observability, failure modes, rollout. Say the trade-off at every box.
Provenance: Generated practice. This is a suggested structure, not a verified interview format for any company.
Activity
Take the six failure patterns above. For each, write the one metric or test that would reveal it, and the first action you would take (alert, degrade, roll back, cap or fix the data).
Homework
Write a one-page release plan for changing the chunker in a multi-tenant RAG service. Include the regression set and its thresholds, the shadow and canary stages, the online metrics that trigger a rollback, and how the cache key is protected from serving answers built from old chunks.
Quick Revision
- Enforce permissions at retrieval and verify on the final context. Retrieved text is untrusted data.
- Key caches on tenant, permissions and document version. Deletion requests must reach vectors, caches and logs.
- Decide in advance what degrades first: drop the reranker before retrieval, and abstain rather than answer without sources.
- Trace every stage and watch RAG-specific dashboards: empty-retrieval rate, abstention rate, ingestion lag.
- Release through offline gate, shadow, canary, rollback. Version prompts, embedder, chunker, index, reranker and LLM.
Check yourself
- Walk-through: A semantic cache serves tenant B an answer built from tenant A’s documents. What went wrong and how do you prevent it?
- Variation: The reranker is slow today. What do you degrade first, and what must you never drop? A canary doubles the abstention rate: what do you do?
- Common misconception: Treating prompt edits as configuration that needs no gate; trusting a passing offline eval as proof of a safe rollout.
Related
Practice
Test what you just read. Quizzes and cards are grouped by module: Production: cost, serving and observability.
Check yourself6 questions on what this lesson set out to teach
Checking answers needs JavaScript. You can still read every question and option below.
Keyboard: arrow keys move between options. Space or Enter selects the focused option.