The core RAG pipeline

Story

Hide story

Merrow slides three pages across the counter to Tobias, who has the microphone and a long run of announcements ahead of him.

Tobias takes the top page, clears his throat, and starts to read. The page is tidy and stamped. It is also headed with last summer’s dates, and nobody has looked at that line.

Between the card drawer and the intercom, which step was supposed to notice?

Concept#

A RAG pipeline is two lanes that meet at a vector index. The offline lane prepares your documents once (parse, chunk, embed, index). The online lane runs on every question (retrieve, rerank, generate).

RAG architecture and chunking introduced the shape. This lesson goes stage by stage, with the decision and the cost at each one.

Why it matters#

“Answer quality is capped by retrieval quality” is the sentence to keep. The generator can only work with what the earlier stages hand it. When an answer is wrong, you need to know which stage to open first, and you can only do that if you know what each stage is for.

What separates strong candidates in interviews is rarely naming the stages. It is evaluating retrieval separately from generation, controlling hallucination, stating latency and cost trade-offs, and knowing when not to use RAG.

Mental model#

Think of a library with a reference desk. Librarians file books in advance (offline lane). When someone asks a question, a clerk fetches candidate books, a specialist narrows them to the best few, and a writer composes the answer using only those books (online lane).

Each person can fail differently. A mis-filed book, a clerk who brings the wrong shelf, a specialist who ranks badly, or a writer who ignores the open book are four different bugs with four different fixes.

Why RAG instead of fine-tuning or long context#

You have three ways to get facts into an answer. Each has a different cost.

• Fine-tuningFine-tuningUpdating a model's weights on labelled examples to teach a consistent behaviour, style or format. It is a poor way to add or update facts.Open the glossary entryClose changes behaviour: style, format, task skill. It is a poor way to inject facts. Knowledge goes stale as soon as the data changes, there are no citations, and you cannot reliably delete one user’s data from model weights.
• Long context (put everything in the prompt) gets more expensive and slower with every token. Accuracy can drop when the prompt is full of irrelevant text, and many corpora are larger than any window.
• RAG gives fresh, citable, permission-aware answers that are cheap to update. The price is that you now run a retrieval system, and answer quality is capped by retrieval quality.

A one-line rule: RAG for knowledge, fine-tuning for behaviour, and they can be combined. If the whole corpus is small and stable, long context may be simpler. See Fine-tuning vs RAG vs prompting for the full decision.

The two lanes#

Diagram: The offline lane (parse, chunk, embed, index) runs ahead of time and fills the vector index.

Text summary

This tool walks one question through a RAG pipeline in six steps: question, embed, retrieve, rerank, prompt and answer (generate). It uses six made-up passages from an imaginary library, and the question is “How do I renew a book online?”. No model runs and nothing is fetched. The passages were embedded ahead of time; at the embed step only the question becomes a vector, with the same embedding model. The vectors have four numbers each and are hand-made, and the cosine similarity scores are computed exactly from them. Retrieval here is dense (vector) retrieval only, to keep it simple. At the retrieve step, the passage about renewing a library card (P1) scores 0.926 and the passage about renewing a book online (P2) scores 0.855, so the wrong one comes first. Retrieve keeps the top three: P1, P2 and P3. At the rerank step, made-up scores from a pretend reranker (often a cross-encoder in real systems), on a 0 to 1 scale, put P2 first (0.91) and P1 second (0.22), so the two swap places. That is why a pipeline reranks: cheap vector similarity finds candidates, and a slower check that reads the question and passage together orders them. The cut from three passages to two is illustrative; real pipelines often shortlist about 50 and keep about 5. The prompt keeps the top two passages with their ids. The answer is written by hand and cites P2. Scores are shown in tables, not by colour alone.

The interactive version needs JavaScript. The text summary above says what it shows.

Stage-by-stage decisions#

RRF in one line#

RRF scores each document as the sum over result lists of 1 / (k + rank). It uses ranks only, so you never normalise the incompatible score scales of BM25 and cosine similarity. The constant k = 60 comes from the original paper (Cormack, Clarke and Buettcher, SIGIR 2009), where it was fixed in a pilot study rather than tuned. It is a sensible default, not a law.

A map of RAG variants#

The simple pipeline above is “naive” RAG plus rerank. Many named variants add a step to fix a specific failure. You do not need them all up front; All RAG types covers when each earns its place.

Diagram: RAG variants fall into three groups: architecture generations from naive to agentic, retrieval-side variants (hybrid, HyDE, parent-child,...

Provenance: common industry patterns. Names and boundaries vary across papers, frameworks and company docs.

Real-world application#

A production assistant adds the unglamorous parts around this pipeline: index versions, a re-ingest schedule, metadata filters, logging of the retrieved chunk ids for every request, and a fallback when retrieval finds nothing. See Production deployment of RAG.

Interview perspective#

Provenance: common industry pattern; no company-specific claim.

The usual prompt is “walk me through a RAG system and where it can fail”. Answer in the order of the stages, give one trade-off per stage, and end with how you would measure it. State the RAG-vs-fine-tuning rule in one sentence early; it signals judgment.

Practice questions#

Provenance: generated practice.

• Basic: Why must queries and chunks use the same embedding model?
• Intermediate: Why is a cross-encoder used only on a shortlist?
• Advanced: A user asks about an error code and dense retrieval returns unrelated pages. What do you change?
• Product company: When would you refuse to use RAG?

Check question#

You are on call. A user asks the assistant a question and gets a confident wrong answer, although you can confirm the correct passage exists in the corpus.

1. List every stage where the failure could originate.
2. Describe the test that tells a retrieval failure from a generation failure.

Worked answer#

1. Ingestion: the passage was parsed badly, split across chunks, or the index holds an older version of the page. Retrieval: wrong embedding or query, a filter (date, tenant) that excluded it, or it ranked below the cut-off. Ranking and context: reranking dropped it, or trimming to the token budget cut it. Generation: it was in the prompt, but the model ignored or misread it.
2. Take the failing query and paste the correct passage into the prompt by hand. If the answer is now right, the passage never arrived: fix retrieval or ingestion. If it is still wrong, fix the prompt or the model. Then log the retrieved chunk ids with their scores, and the final prompt, on every request, so you can check “was it retrieved?” without guessing.

Activity#

Do, notice, apply.

1. Do. Pick any small set of documents you can read (five pages of a manual or wiki). Write one question whose answer is on a single page, and one whose answer needs two pages.
2. Notice. For each question, trace it through the stages: which page would a keyword search find? Which would a paraphrase-matching search find? Where could each fail?
3. Apply. Add a sixth page to the set, an older copy of one page with different numbers. Which stage must prevent the old copy from being used, and what metadata would it need?

Homework#

Answer the check question in writing, then aloud without notes. Compare your list of failing stages with the worked answer and mark any stage you missed.

Quick revision#

• RAG for knowledge, fine-tuning for behaviour. Long context is costly and noisy at scale.
• Offline lane: parse, chunk, embed, index. Online lane: retrieve, rerank, generate.
• Query and chunks share one embedding model; changing it means re-indexing.
• Hybrid with RRF (k = 60 by default) fixes exact-token misses without score normalisation.
• Rerank a wide shortlist down to a few.
• Separate retrieval from generation by pasting the gold passage into the prompt.

Check yourself#

• Checkpoint (answer aloud, no notes): Explain every stage and its trade-off in under three minutes.
• Re-test (nearby variant): Users report outdated answers after a policy change. Which stages do you inspect, in what order?
• Misconception to avoid: “A better model fixes bad retrieval.” The model never sees chunks that were not retrieved.

Related#

• Better retrieval
• All RAG types
• RAG evaluation in depth
• Production deployment of RAG
• RAG interview drills and mock