Designing a RAG QA system

Story

Hide story

Dawn at the Far Capital. A clerk from the Network Office slides a blank sheet across the desk. At the top it says: a timetable assistant for forty stations. Draw it.

Tobias suggests starting with a better microphone. Merrow suggests starting with the drawers. Neither has said who may open which drawer.

You pick up the pencil. What do you ask before you draw the first box?

Every number in this lesson is illustrative. The corpus, the user counts and the prices are assumptions chosen so the arithmetic is easy to follow. In an interview, say “assume” and show the multiplication. Never present such numbers as facts about a real product.

Concept#

A system-design answer is a chain of decisions, each justified by a requirement or a number. For a GenAI product the chain has nine links: requirements, scale estimate, components, data flow, failure modes, evaluation, cost and latency, security, rollout.

This lesson teaches that chain on the most common GenAI design prompt: a question-answering assistant over a company’s internal documents, where each employee may only see what they are allowed to see. The other lessons in this module reuse the same chain for agents, moderation and a timed mock.

Why it matters#

A design interview, and a real design review, rewards the same thing: reasoning from constraints to choices. A list of components (“vector database, reranker, LLM”) shows that you have read about RAG. A chain of reasons (“2 million chunks fit in memory on one node, so I do not shard yet”) shows that you can build one.

The nine links also expose the details that rushed answers skip: who may see which document, what the system does when it does not know, and how you would find out that it is wrong.

Mental model#

Think of a reference librarian at a members-only library. A member asks a question. The librarian checks which shelves this member may enter, searches only those shelves, brings back a few passages, and writes an answer that points to the pages. If nothing relevant is on the allowed shelves, the librarian says so rather than guessing.

Each part of the system is one duty of that librarian: the permission check, the search, the shortlist, the cited answer and the honest “I did not find it”.

The nine links, as questions you ask in order:

Requirements#

Start by asking, then state your assumptions aloud. For this design:

• Functional: answer questions grounded in internal documents (wiki pages, PDFs, tickets) and cite the source passage.
• Access control: a user must never receive an answer built from a document they cannot open.
• Freshness: an edited document should be searchable within a stated time. Assume 15 minutes for this lesson.
• Latency: assume the first token appears within 3 seconds at p95, with the answer streamed.
• Quality bar: a wrong answer is worse than no answer, so the system may abstain.
• Out of scope: taking actions in other systems, and documents in languages other than English.

Writing “out of scope” is part of the answer. It shows you can bound a problem.

Scale estimate with worked numbers#

Estimate the numbers that change a decision. Skip the rest.

Assumptions: 5,000 employees, 200,000 documents, about 10 chunks per document after splitting, a 1,024-dimension embedding stored as 4-byte floats.

Now tokens and cost. Assume a prompt of 500 tokens of instructions, 5 retrieved chunks of 500 tokens, and a 100-token question: 500 + 2,500 + 100 = 3,100 input tokens, plus 300 output tokens. Assume a hypothetical price of $3 per million input tokens and $15 per million output tokens.

• Input: 3,100 × $3 / 1,000,000 = $0.0093.
• Output: 300 × $15 / 1,000,000 = $0.0045.
• One query: $0.0093 + $0.0045 = $0.0138, about 1.4 cents.
• One day: 7,500 × $0.0138 = $103.50.
• One month of 22 working days: 165,000 queries × $0.0138 = $2,277, about $2,300.

Three lessons fall out of the numbers.

1. The index is small and the traffic is low. About 1 query per second does not need a distributed vector database. Over-building here is a design flaw, not a strength.
2. The LLM call is the cost. Retrieved chunks are 2,500 of the 3,100 input tokens. Dropping from 5 chunks to 3 gives 500 + 1,500 + 100 = 2,100 input tokens, so one query costs 2,100 × $3 / 1,000,000 + $0.0045 = $0.0063 + $0.0045 = $0.0108, a saving of about 22%. That is a quality trade, so measure it (Cost and latency optimization).
3. Re-embedding is cheap, so freshness is an engineering problem, not a cost problem. Embedding all 2,000,000 chunks of about 500 tokens is 1 billion tokens; at an assumed $0.10 per million that is $100 once. If 2% of documents change each day, that is 4,000 documents, 40,000 chunks, 20 million tokens, or about $2 per day.

State the numbers, then say what would change them. At 50 million chunks the raw vectors are 50,000,000 × 1,024 × 4 = 204.8 GB, or about 300 to 400 GB with index overhead at the same 1.5 to 2 times assumption, which is beyond a typical node’s memory. Then you shard, quantize vectors, or use a disk-based index (Vector databases).

Components and data flow#

The system has two paths that meet at the index.

Diagram: Documents and their permissions flow through load, chunk, embed and store, with each chunk carrying its allowed groups.

Walk the query path in words:

1. Resolve who the user is and which groups they belong to, from your identity system, not from the request body.
2. Embed the question and run a hybrid searchHybrid searchRunning keyword search (BM25) and vector search together and merging the lists. It covers exact terms and paraphrases.Open the glossary entryClose (vector plus keyword), with the permission filter applied inside the query.
3. Rerank the top 50 candidates down to 5 with a cross-encoderCross-encoderA model that reads query and passage together and outputs a relevance score. More accurate, but one call per pair, so it reranks a shortlist.Open the glossary entryClose.
4. If the best reranked score is below a tuned threshold, stop and say nothing was found.
5. Otherwise build the prompt, ask for an answer that cites chunk ids, and stream it.
6. Log the query, the chunk ids, the scores, the answer and the cost.

Why these choices, with their costs:

• Hybrid retrieval. Vector search matches meaning; keyword search matches exact tokens such as error codes and product names. Pure vector search can miss the exact-term queries, so hybrid is a common default (Better retrieval). The cost is a second index to keep in sync.
• RerankingRerankingRe-ordering a shortlist of retrieved candidates with a more accurate model. It cannot recover a chunk the first stage failed to return.Open the glossary entryClose. A cross-encoder reads the query and each candidate together, which is more accurate than the first-stage embedding match but too slow for the whole corpus. The cost is added latency on every query.
• Citations. They let a user verify, and let you measure whether cited passages support the claims. The cost is prompt complexity and a check that the cited ids were really retrieved.
• Abstain pathAbstentionA system declining to answer, or asking a clarifying question, when retrieval is empty or weak. Calibrate the gate on labelled data.Open the glossary entryClose. The cost is some answerable questions refused. Set the threshold on a golden set that includes unanswerable questions (RAG evaluation in depth).

Access control inside retrieval#

This is the detail rushed answers skip, so give it its own section.

Rule: the permission filter is part of the retrieval query. Only chunks the user may read can ever reach the prompt. The model is never asked to hide something it was shown, because a model that has seen a secret can be argued into repeating it (Limiting blast radius and treating output as untrusted).

How to build it:

• Each chunk stores allowed_groups (or user ids) copied from its parent document at ingest time.
• At query time, the user’s group list goes into the search as a metadata filterMetadata filterA condition on fields such as department, date or permission applied to a vector search. Applied after the search it can return fewer than k results. See post-filtering.Open the glossary entryClose. Many vector databases support filtered search, but filtered approximate search has recall and latency behaviour that you must test on your own index.
• Permissions change after ingestion. Someone leaves a group or a document becomes private. So run an event-driven permission sync from the source of truth, set a stated lag target, and re-check the final 5 chunks against the source system before the prompt is built. Five checks per query is cheap insurance against sync lag.
• Caches obey identity. A semantic cache that returns user B’s answer to user A is an access-control bug. Key any answer cache on the permission set, or cache only the stages that do not contain restricted text (Cost and latency optimization).

What goes wrong if you filter after generation: the model has already read the restricted passage. The answer may paraphrase it, and a post-hoc checker cannot reliably recognise a paraphrase of text it is not allowed to see. Filtering the retrieved list after the search but before the prompt is safe from leaks, but it can leave you with fewer than 5 chunks, or none, because the top-k was spent on documents the user could not read. That is a recall problem, which filtering inside the query avoids.

Evaluation plan#

Reuse the method from RAG evaluation in depth rather than inventing a new one.

• Before launch: build a golden set of real questions from subject experts, each with gold passages, including unanswerable questions and questions whose answer sits in a restricted document.
• Retrieval metrics: recall@k and MRRMRRShort for mean reciprocal rank: the average of 1/rank of the first relevant result, counting 0 for a miss.Open the glossary entryClose against the gold passages, run separately for each permission group.
• Generation metrics: faithfulness to retrieved context, answer correctness and citation accuracy, with a judge whose bias you have checked (Evaluating LLM systems).
• Security tests: a fixed set of “user A asks about user B’s document” probes. The expected result is zero leaked passages.
• After launch: thumbs up or down, abstention rate, follow-up rephrasing, and a weekly sample of traces reviewed by a person. Failures join the golden set.

Say why the split matters: when quality drops, the stage metrics tell you whether to fix ingestion, retrieval or generation.

Failure modes#

Name the failure, the signal that detects it, and what the system does meanwhile.

Production context: the stale index and the permission lag are the two failures users cannot see, so they need alarms rather than user complaints.

Cost, latency and rollout#

Latency budget (illustrative, p95 target 3 seconds to first token): auth and query embedding 100 ms, retrieval 100 ms, rerank 300 ms, LLM time to first token 1,200 ms. That sums to 100 + 100 + 300 + 1,200 = 1,700 ms, leaving 1,300 ms of margin for network and queueing. Stream the answer so the user perceives the first token, not the last. Levers if over budget: a smaller reranker, fewer candidates, a smaller model for easy questions, prompt caching of the static prefix (Cost and latency optimization).

Rollout: run offline evaluation, then shadow traffic, then a small canary, with every component versioned so you can roll back. A change to the chunker, embedder, prompt or model passes the same gate (RAG security, reliability and release).

Key trade-offs to say aloud#

• Chunk size: small chunks are precise but lose context; large chunks carry context but dilute the match and raise token cost.
• Freshness versus complexity: near-real-time sync is more machinery than nightly batch. Ask what the users need.
• Hybrid versus vector only: better exact-term recall for a second index.
• Abstain threshold: a stricter threshold means fewer wrong answers and more refusals.
• Rerank depth: more candidates improve recall at the price of latency.

Interview perspective#

Provenance: common industry pattern; no company-specific claim.

Interviewers commonly look for requirements before components, numbers that drive choices, permission filtering inside retrieval, an abstain path, a split evaluation, and failure modes raised without a prompt. The permission step and the abstain path are the two details that rushed answers often skip. How a particular interviewer weighs them is judgment, not data.

Practice questions#

Provenance: generated practice.

• How would you make sure user A never sees an answer grounded in a document only user B can open?
• Why might hybrid search beat vector-only search for this use case?
• Re-index frequency: how would you decide it for frequently updated tickets and for static policy documents?
• At 50 million chunks and 500 queries per second, what breaks first, and what changes?

Activity#

Work through the design on paper in 25 minutes.

1. Take the assumptions from the scale section and change two of them (for example 20,000 employees and 1,000 tokens per chunk). Recompute chunks, raw vector size, tokens per query and monthly cost by hand.
2. Write one sentence for each number: which decision it drives.
3. Draw the permission step in words. Where does it sit, and what happens if you apply it after generation?
4. Reflect: which of your numbers would you most want to measure in the real system before trusting it?

Homework#

Write a one-page design document for a question-answering system over your team’s own (hypothetical) documentation. It needs: requirements with out-of-scope items, a scale estimate with labelled assumptions, the two-path diagram, the permission design, an evaluation plan, a failure table and a rollout plan. Check each number by recomputing it.

Quick revision#

• Nine links: requirements, scale, components, data flow, failures, evaluation, cost and latency, security, rollout.
• Estimate only the numbers that change a decision, and label them illustrative.
• Filter permissions inside the retrieval query, re-check the final chunks, and scope caches by identity.
• Abstain below a tuned evidence threshold.
• Evaluate retrieval and generation separately, with unanswerable and cross-permission test cases.
• Every failure gets a detection signal and a degraded mode.

Check yourself#

• Checkpoint question (answer aloud, no notes): Whiteboard a RAG QA system for 5,000 employees with per-document permissions. Give numbers for index size and cost per query, and your evaluation plan.
• Independent re-test: Scale to 50 million chunks and 500 queries per second. What breaks first, and what changes?
• Common misconception to avoid: Listing components before asking requirements, applying access control after retrieval or generation, and shipping without a split evaluation.

Related#

• RAG architecture and chunking
• RAG evaluation in depth
• Production deployment of RAG
• RAG security, reliability and release
• Designing an agent product