RAG interview drills and mock

Story

Hide story

The board sits at the long table in the Network Office: four people and a pot of tea. This morning’s sheet is pinned to the wall, now covered in boxes and arrows, with a short list in the margin, one line for each member of the crew.

“Walk us through it,” the chair says. “Start where it breaks.”

You take a breath. What do you say first, and what do you say you do not know?

Every number in the worked answers is illustrative. In a real interview, say “assume” and show the arithmetic.

Concept#

A drill is a short, repeatable question that trains one skill under time pressure. A mock is a full interview run end to end with someone else asking the questions. This lesson turns what you learned in Designing a RAG QA system and the rest of the course into both: a question bank, a 45-minute mock structure, a rubric and worked answers. The next lesson, Coding, project defense and mock rounds, covers the coding drills, defending your own project and a three-round mock plan. Each lesson stands alone.

Why it matters#

Knowing RAG and explaining it aloud in 90 seconds are different skills. Under time pressure, people skip requirements, forget trade-offs, and name components without reasons. Drills train fluency on the parts. A mock trains the sequencing, the pacing and the recovery when the interviewer pushes back.

Practice only helps if you can tell what went wrong. That is why this lesson includes a rubric: it turns “that felt okay” into named evidence you can fix.

Mental model#

Think of a flight simulator. A pilot does not learn emergencies by waiting for one. The simulator scripts a normal flight, then injects faults on purpose: an engine warning, a crosswind. A good mock does the same. The interviewer sets a normal design problem, then changes a constraint (“now 500 queries per second”, “a customer saw another team’s document”) to see how you adapt.

So you practise two things: the normal flight (the nine-link design chain) and the fault drills (debugging and constraint changes).

The 45-minute design mock#

Use this split. It follows the chain from the first lesson in the module: requirements, scale, components, data flow, failure modes, evaluation, cost and latency, security, rollout.

The blocks sum to 5 + 5 + 15 + 13 + 5 + 2 = 45 minutes. In a real interview the interviewer may jump ahead; follow them, but return to the skipped blocks.

Diagram: The mock moves from requirements through scale, architecture and deep dives to failures and a summary within 45 minutes.

Handling a constraint change: restate the new constraint, say which earlier number it changes, recompute that number, then say which component changes and what it costs. For example, “500 queries per second” changes the peak rate you estimated, so you recompute the peak rate, then the replica count, LLM rate limits and LLM cost before you answer; index size is unchanged, because it depends on the number of chunks, not on traffic.

Scoring rubric#

A partner, a mentor, or you reviewing a recording scores each dimension from evidence in the answer, not from impression. Where torn between two levels, take the lower and name the evidence that would earn the higher.

After scoring, pick the weakest dimension and run one drill for it before the next mock. A weak “numbers” score calls for the scale practice in the first lesson. A weak “failure modes” score calls for the debugging drills below.

Drill A: rapid-fire concept questions#

Provenance: generated practice; the topics are common industry patterns. No company-specific claim. Aim for 60 to 90 seconds each, with one trade-off.

1. What is RAG and why use it over fine-tuning? Retrieve evidence at query time and ground the answer in it. It is fresh, citable, permission-aware and cheap to update. Fine-tuning mainly changes behaviour and style, and is a poor way to keep facts current (Fine-tuning vs RAG vs prompting).
2. When would you not use RAG? A small static corpus that fits in the context (just prompt it), pure reasoning or formatting tasks, structured questions better served by SQL or an API, and cases where retrieval adds latency for little benefit.
3. Chunk size trade-off? Small chunks are precise but lose context; large chunks carry context but dilute the match and cost more tokens. Tune on an evaluation set, with overlap and structure-aware splits.
4. Dense, sparse or hybrid? Dense captures meaning, sparse matches exact tokens, hybrid with rank fusion covers both.
5. Bi-encoder versus cross-encoder? A bi-encoderBi-encoderA model that embeds query and document separately, so document vectors can be precomputed. Fast, used for first-stage retrieval.Open the glossary entryClose embeds query and document separately (fast, indexable); a cross-encoderCross-encoderA model that reads query and passage together and outputs a relevance score. More accurate, but one call per pair, so it reranks a shortlist.Open the glossary entryClose scores the pair jointly (more accurate, slow), so use it to rerank a shortlist.
6. Why approximate nearest neighbour search? Exact search compares the query to every vector (O(N) per query). HNSW and similar indexes trade a small amount of recall for large speed gains (Vector databases).
7. Why does RAG still hallucinate? Missing or wrong retrieval, noisy context, the model ignoring context, ambiguous sources, and no instruction to abstain.
8. How do you reduce hallucination? Better retrieval, a grounding prompt with citations, abstention on low evidence, faithfulness checks, smaller focused context, and output verification.
9. What is “lost in the middleLost in the middleThe finding that models use information in the middle of a long context less reliably than information at the start or end.Open the glossary entryClose”? Models can use information at the start and end of a long context better than the middle (Liu et al., listed in resources.md), so order and trim.
10. What is HyDEHyDEHypothetical document embeddings: the model writes a plausible answer and that text is used to search. It helps when queries and documents are phrased differently, but a wrong draft can bias retrieval.Open the glossary entryClose and its risk? Embed a hypothetical answer instead of the question; the risk is bias toward a wrong hypothetical.
11. GraphRAG versus vector RAG? A graph helps multi-hop and global questions and costs more to build; vector retrieval suits local lookups.
12. Self-RAG, CRAG, adaptive RAG? Self-RAG has the model critique itself with reflection tokens; CRAG grades retrieval and falls back; adaptive routing picks a strategy by query complexity (All RAG types).
13. Parent-child retrieval? Match on small chunks, return the larger parent for context.
14. Why metadata filtering? Precision, freshness, tenancy and access control, applied inside the retrieval query.
15. How to choose an embedding model? Domain and language fit, dimension, latency, cost, maximum length and licence; evaluate on your own data, not leaderboards alone.
16. What happens when the embedding model changes? You must re-embed the corpus, because queries and documents must share one model. Version the index and swap with an alias.
17. How do you handle multi-turn chat? Rewrite the follow-up into a standalone query, budget the history, and skip retrieval on small talk.
18. How do you evaluate RAG? Retrieval (recall@k, MRR, nDCG) separately from generation (faithfulness, relevance, correctness), plus online signals (RAG evaluation in depth).
19. Context windows are huge, is RAG dead? No. Cost and latency scale with tokens, relevance can dilute in long contexts, corpora can exceed any window, and permissions, citations and freshness still need retrieval. Long context and RAG are often combined.
20. Prompt injection through documents? Treat retrieved text as untrusted data, delimit it, limit tool permissions, filter outputs and sanitise sources (Guardrails, safety and prompt injection).

Drill B: debugging scenarios#

Provenance: generated practice. Answer in a fixed order: restate, hypothesise by stage, name the first measurement, then the fix.

1. “The bot answers confidently but wrong, and the right document exists.” Follow the debug tree in Better retrieval: was it indexed, was it retrieved, was it in the final prompt, did the model use it? A paste-the-gold-passage test separates retrieval from generation.
2. “Quality dropped after last week’s deploy.” Diff the versions (chunker, embedder, prompt, model, index), run the regression set, check ingestion, and roll back through the alias.
3. “Users search by product code and results are irrelevant.” Add keyword or hybrid search, an exact-match boost, and a metadata filter.
4. “Answers are slow, p95 is 8 seconds.” Trace each stage, stream, cut candidates, use a smaller reranker, cache, and route easy queries to a smaller model.
5. “Cost doubled.” Check tokens per request, top-k and chunk size, agent loops, cache hit rate and model routing.
6. “A customer saw another team’s document.” Treat it as an incident: check the ACL filter, cache keys and index namespaces, then add tests and an audit.
7. “Great on the evaluation set, bad in production.” The set is probably unrepresentative, dominated by synthetic questions, or overlapping with tuning data; refresh it from production logs and slice metrics.

Drill C: system design prompts#

Provenance: generated practice. No company asks these on our authority. Each is a 45-minute mock using the split above.

1. An enterprise knowledge assistant over Confluence, Drive and Jira with per-user permissions.
2. A customer-support bot with citations and human handoff, 1 million tickets and daily updates.
3. A legal or contract assistant (long documents, cross-references between clauses).
4. A code-search assistant over a monorepo.
5. A multimodal assistant over financial reports (tables and charts).

For the production checklist behind these prompts see Production deployment of RAG and RAG security, reliability and release.

Worked answers#

Three model answers, shown with their reasoning. Do your own version first, then compare.

1. “When would you not use RAG?” (60 to 90 seconds)#

“RAG earns its cost when the knowledge is large, changing, or needs citations and permissions. I would skip it in three cases. First, if the whole corpus fits comfortably in the context and rarely changes, I would put it in the prompt, which removes the retrieval stage and its failure modes. Second, if the question is structured, like ‘total sales last quarter’, I would use SQL or an API, because retrieval over text would be slower and less exact. Third, if the task is reasoning or formatting with no external knowledge, retrieval adds latency for nothing. The trade-off I would check is cost: long prompts cost more per query than retrieving five chunks, so ‘fits in context’ is not enough on its own.”

Why it works: three cases, one trade-off, one measurable consideration.

2. “Answers are slow, p95 is 8 seconds.” (debugging)#

First measurement: trace every stage. Suppose the trace shows (illustrative): query embedding 0.1 s, retrieval 0.3 s, rerank 1.1 s, LLM time to first token 2.0 s, and 4.5 s to generate the answer without streaming. Total: 0.1 + 0.3 + 1.1 + 2.0 + 4.5 = 8.0 s.

• Fix 1, stream the answer. The user now waits for the first token only: 0.1 + 0.3 + 1.1 + 2.0 = 3.5 s to first token. Total time is unchanged; perceived latency drops.
• Fix 2, rerank 20 candidates instead of 50. If rerank time scales roughly with candidates, 20/50 × 1.1 s is 0.44 s, call it 0.5 s. Time to first token becomes 0.1 + 0.3 + 0.5 + 2.0 = 2.9 s. The cost is lower recall if the right passage sat between ranks 21 and 50, so check recall@20 on the golden set.

Why it works: measurement first, arithmetic, and the cost of the fix named. See Cost and latency optimization.

3. “Cost doubled.” (debugging)#

Say what you would check first and show that you can test a hypothesis. Suppose someone raised top-k from 5 to 10 with 500-token chunks. Input tokens go from 500 + 2,500 + 100 = 3,100 to 500 + 5,000 + 100 = 5,600, an increase of 81%. At a hypothetical $3 and $15 per million tokens with 300 output tokens, cost per query goes from $0.0138 to 5,600 × $3 / 1,000,000 + $0.0045 = $0.0168 + $0.0045 = $0.0213, an increase of about 54%.

So the top-k change alone does not explain a doubling. Look next at cache hit rate, an agent loop or retry storm, a model-routing change, and a traffic increase.

Why it works: it tests the obvious hypothesis with numbers and shows it falls short.

Interview perspective#

Provenance: common industry pattern; no company-specific claim.

Design rounds for GenAI roles commonly look for requirements before components, numbers tied to decisions, separate retrieval and generation evaluation, failure modes raised without a prompt, and honest answers about what you have and have not measured. The exact format and weighting vary by company and team, and this course makes no claim about any specific one.

Activity#

A 45-minute solo or paired session.

1. Run a 5-minute timer and answer five questions from Drill A aloud, recording yourself.
2. Pick one scenario from Drill B and answer in the fixed order (restate, hypothesise by stage, first measurement, fix).
3. Do the first 15 minutes of the 45-minute mock on one prompt from Drill C (requirements, scale and the start of the architecture block). Run the full 45 minutes as a separate session.
4. Score the recording against the rubric: for each dimension, write the evidence (a quote or a missing item), not an adjective.
5. Choose the weakest dimension and write one drill for it.

Homework#

Write your own worked answer, in the format above, for Drill B scenario 6 (a customer saw another team’s document). Include the first measurement, three hypotheses by stage, the fix, a test that would catch a repeat, and what you would tell the customer. Then schedule a second mock on a different prompt and compare your rubric scores.

Quick revision#

• Drills train parts; a mock trains sequencing, pacing and recovery.
• The 45-minute split: 5, 5, 15, 13, 5, 2 minutes.
• On a constraint change: restate, find the affected number, recompute, name the component and the cost.
• Score with evidence per dimension; improve the weakest one first.
• Debugging order: restate, hypothesise by stage, first measurement, fix with its cost.

Check yourself#

• Checkpoint question (answer aloud, no notes): Run a 45-minute mock design for a customer-support RAG bot with 1 million tickets.
• Independent re-test: The interviewer says traffic is now 100 times higher. Which number do you recompute first, and what changes?
• Common misconception to avoid: Reciting a component list without reasons, and scoring yourself on feeling instead of evidence.

Related#

• Designing a RAG QA system
• Designing an agent product
• Designing content moderation and guardrail systems
• Coding, project defense and mock rounds
• Production deployment of RAG
• RAG security, reliability and release