Production deployment of RAG

Story

Hide story

The telegram arrived at midnight: the 02:40 connection is cancelled, revised timetable attached. The night clerk signed for it and left it on Merrow’s desk. Merrow was two carriages down.

The strips in the drawers still show the 02:40. Just after one, Tobias reads it to a full train, warmly, from the page in front of him.

Who was supposed to tell the drawers?

Numbers below are illustrative. State your assumptions in interviews and never present them as facts.

Concept#

A RAG prototype is a notebook: embed documents, retrieve, prompt, answer. A production RAG system is that pipeline plus everything that keeps it correct, fast, affordable, secure and changeable while real users and real documents move: a gateway, permission checks, caches, an ingestion pipelineIngestionThe offline part of a RAG system: parse, chunk, embed and index documents. Safe ingestion is idempotent, incremental and records model and chunker versions.Open the glossary entryClose, observability and a release process. The retrieval and generation ideas come from The core RAG pipeline and RAG evaluation in depth. This lesson covers the shape of the system: its architecture, ingestion, freshness, latency, cost and scaling. The next lesson, RAG security, reliability and release, covers permissions, failure handling and safe change.

Why It Matters#

Many production RAG failures have nothing to do with the model: the index goes stale, an embedding change collapses recall, latency creeps past its target, or cost doubles with no explanation. Interviewers ask for a whiteboard design precisely because these are the questions that separate a demo from a service. The design is mostly about trade-offs at the boundaries, so say the trade-off at every box.

Mental Model#

A production RAG system is a library with two separate rooms.

The reading room (read path) answers visitors quickly and must never show a book to someone who may not see it. The back office (write path, ingestion) catalogues new and changed books, removes withdrawn ones, and prepares updated shelves. The index is the shelving that joins them. You never rearrange shelves while visitors are browsing: you build the new shelving in the back office, test it, and swap the signs (an aliasIndex aliasA name that points to the live index version. Swapping it switches versions at once and allows one-step rollback.Open the glossary entryClose) in one move, keeping the old shelves ready to swap back.

Reading room needs speed and access control. Back office needs correctness, freshness and versioning. Most design questions are about which room a problem belongs to.

Reference architecture#

Diagram: A request passes the gateway, input guardrails and query transform, then checks a cache.

Each box has a typical failure: the gateway rate-limits wrongly, guardrails add latency or block valid input, the cache serves a wrong or cross-tenant answer, retrieval misses or leaks, the reranker is slow, the LLM provider times out, and output checks conflict with streaming (you cannot stream text you have not yet checked, so choose between checking first, streaming first, or checking in chunks).

Ingestion and indexing pipeline#

Diagram: Changes in source systems are detected, queued, parsed, chunked with metadata and permissions, embedded with a versioned model and written...

Key properties:

• Idempotent: re-running gives the same chunks and idsDeterministic chunk IDA chunk id that comes out the same each time the same input is ingested, such as the document id plus the chunk number or a hash of the chunk's text. Re-running ingestion then updates chunks instead of duplicating them. Old chunks of an edited or shortened document still have to be deleted, because a changed chunk can get a new id.Open the glossary entryClose (use content hashes), so retries do not duplicate.
• Incremental: only changed documents are reprocessed.
• Deletes propagate: removed or revoked documents leave the index (tombstonesTombstoneA marker recording that a document was deleted, so the deletion reaches the index and the content stops being retrieved.Open the glossary entryClose), otherwise they keep being retrieved.
• Versioned: record the embedding model and chunker version on every chunk.

Freshness and re-embedding#

Define the promise (“new documents are searchable within N minutes”) and monitor ingestion lag against it. Use incremental upserts for normal changes. For a big change, such as a new embedding model, build a complete new index version in the background, evaluate it, then switch the alias (blue/green). Queries and documents must share one embedding model, so the switch must move both together. Keeping the old version for a while makes rollback one alias change.

Latency budget#

Example target: p95 of about 3 s to first token, streaming. An illustrative allocation:

The rows sum to 3.00: 0.40 + 0.15 + 0.35 + 0.05 + 1.60 + 0.45. Percentiles do not add exactly, so use the table to plan and measure the end-to-end p95 directly. Stream tokens, and run independent calls (dense and keyword search, multiple query variants) in parallel. See Cost and latency optimization for the levers.

Cost per query#

Cost per query is embedding + retrieval + rerank + LLM input and output tokens, tracked per tenant. Tokens often dominate, so the first levers are fewer and shorter chunks, prompt trimming, routing and prompt caching.

Illustrative arithmetic with the prices from the previous lessons (2 dollars per million input tokens, 10 per million output): five chunks of 500 tokens, a 500-token system prompt, a 50-token question and a 300-token answer is 3,050 input tokens and 300 output tokens. Input costs 3,050 x 2 / 1,000,000 = $0.0061 and output 300 x 10 / 1,000,000 = $0.0030, so $0.0091. Cutting to three chunks gives 2,050 input tokens: $0.0041 + $0.0030 = $0.0071, about 22 percent less. Whether recall survives is an evaluation question.

Caches: an embedding cache, a retrieval-result cache, a semantic cache for near-duplicate questions, and provider prompt caching for the static prompt. A semantic cache must key on tenant, permissions and document version, or it can serve a stale or wrong-user answer.

Scaling the vector layer#

• Capacity: vectors x dimensions x bytes. For 10 million vectors of 768 dimensions at 4 bytes: 10,000,000 x 768 x 4 = 30.72 billion bytes, about 31 GB raw, before index overhead. That decides RAM versus disk, quantization and sharding.
• Shard for size, replicate for query rate and availability, tier hot and cold data.
• Trade-offs: HNSW is fast but memory-heavy. IVF with product quantization or disk-based indexes are cheaper but cost recall or latency. Tune the recall target against latency (for example efSearch or nprobe). See Vector databases.
• Multi-tenancyMulti-tenancyOne system serving many customers whose data must stay separate. Permissions, caches and indexes must be scoped by tenant.Open the glossary entryClose: a namespace per tenant gives isolation and a bigger operational footprint. A shared index with a tenant filter is cheaper but has noisy-neighbour and filter-bug risk.

Deployment topologies#

• Managed: hosted vector DB plus hosted LLM API. Fastest to ship; vendor lock-in and a data-governance review.
• Self-hosted: open-source vector DB and LLM on Kubernetes (GPUs for rerank and LLM). More control and possibly lower cost at scale; real operations burden (LLM inference and serving).
• Hybrid or VPC: private networking to hosted models for sensitive data.

Containerize services, autoscale the stateless orchestrator on request rate, autoscale GPU workers on queue depth, and run ingestion workers separately from serving.

Activity#

Draw the reference architecture from memory. Next to each box, write its typical failure and the one metric that would reveal it.

Homework#

Write a one-page design for zero-downtime re-embedding with an eval gate and rollback. Include the alias swap, how deletions are handled during the rebuild, and the metrics you watch during the canary.

Quick Revision#

• Two paths: read (fast, access-controlled) and write (correct, fresh, versioned).
• Ingestion is idempotent, incremental, delete-aware and versioned. Re-embed into a new index and swap an alias.
• Budget latency per stage; measure the real end-to-end p95. Tokens often dominate cost.
• Size the vector layer with arithmetic (vectors x dimensions x bytes), then shard for size and replicate for query rate.
• Run ingestion workers separately from serving so batch work does not compete with user requests.

Check yourself#

• Walk-through: How do you guarantee a deleted or revoked document stops appearing?
• Variation: Design zero-downtime re-embedding. How would you size the index for 10 million vectors?
• Common misconception: Swapping the embedding model without re-indexing; treating the sum of stage latencies as the end-to-end p95.

Next: RAG security, reliability and release covers permissions, graceful degradation, observability and safe rollout of the system you have just designed.

Related#

• RAG security, reliability and release
• RAG evaluation in depth
• Observability and eval pipelines
• RAG question answering system design