Better retrieval

Story

Hide story

The right timetable page is in the car. You can see its corner sticking out of the drawer.

Merrow has pulled two candidates and finds the second one plausible. “That’ll do,” he says, and reaches for the stamp.

From the counter, how would anyone know whether the best page was ever on the pile?

Concept#

Better retrieval means getting the right chunk into the prompt more often. The tools are: searching two ways and merging (hybrid), re-scoring a shortlist (rerank), fixing the query, filtering by metadata, shaping the chunks, and assembling the final context carefully.

Why it matters#

Answer quality is capped by retrieval quality. Many “the LLM hallucinated” bugs are really “the right chunk never reached the prompt”. Fixing the generator will not help if its evidence is wrong.

Each technique below has a cost in latency, money or complexity. The skill is applying the cheapest fix to the failure you have actually measured.

Mental model#

Retrieval is a two-step search party. First a wide net brings in many candidates cheaply, so the right one is probably in the catch (high recall). Then a careful sorter looks closely at each candidate and keeps the best few (high precision).

Every technique in this lesson widens the net, sharpens the sorter, or cleans up what is handed over.

Hybrid search#

• Dense (embeddings) matches meaning and paraphrase. It is weak on exact tokens such as IDs, error codes and rare names.
• Sparse (BM25BM25A keyword ranking function that scores documents by term frequency, term rarity and length. It catches exact terms and identifiers that embeddings miss.Open the glossary entryClose or keyword) matches exact terms. It is weak on synonyms and paraphrase.
• Hybrid runs both and merges the lists.

Prefer Reciprocal Rank Fusion (RRF)Reciprocal rank fusion (RRF)A way to merge ranked lists by summing 1/(k + rank) per list, with k often 60. It uses ranks only, so incompatible score scales need no calibration.Open the glossary entryClose: score(d) = sum over lists of 1 / (k + rank_i(d)). The default k = 60 comes from the original paper (Cormack, Clarke and Buettcher, SIGIR 2009), where it was fixed in a pilot study. RRF uses ranks only, so you never normalise incompatible score scales. The alternative is a weighted sum of normalised scores, which can do better when tuned but needs tuning and labelled data.

Worked example, k = 60. Document A is rank 1 in the dense list and rank 3 in the BM25 list: 1/61 + 1/63 is about 0.0323. Document B is rank 2 in dense and rank 1 in BM25: 1/62 + 1/61 is about 0.0325. B wins the fused list because it ranks well in both, even though A won the dense list.

Diagram: The query goes to dense search and BM25 in parallel, each returning a top 50.

Reranking#

• A bi-encoderBi-encoderA model that embeds query and document separately, so document vectors can be precomputed. Fast, used for first-stage retrieval.Open the glossary entryClose (used for retrieval) embeds query and chunk separately. It is fast and approximate, and chunk vectors can be precomputed.
• A cross-encoder (used for reranking) reads query and chunk together. It is more accurate but needs one model call per pair, so you run it on the shortlist only.

Pattern: retrieve wide (for example 50 to 100), rerank narrow (for example 3 to 8). The cost is added latency and money in exchange for better precision in the top results. Measure it on your own question set; reranking is not free and not always worth it.

Query transformation#

Fix the query instead of the index.

Every transform adds an LLM call (latency and cost) and can drift from the user’s intent. Gate them: apply them only when a quick check says they are needed. Multi-query, for instance, hurts when the query is already precise, because the paraphrases dilute it.

Metadata filtering and permissions#

• Store source, date, document type, language, tenant and ACL as metadata.
• Filter before or during the vector search (pre-filter), not after.
• Post-filtering a top-k can leave you with zero results, because the best matches were the ones removed.
• Pre-filtering can hurt recall or latency in some approximate indexes. Know how your database behaves.
• Take the filter value (tenant, group) from the authenticated identity on the server, never from text the user supplied. Some stores offer a namespace per tenant, which makes the restriction part of the search itself.
• Permissions are enforced in the retriever. Asking the LLM to hide things is not a control, because the text is already in the prompt.

Chunk-level improvements#

• Parent-child (small-to-big): match on small chunks, return the larger parent.
• Sentence window: match a sentence, return it with N neighbours on each side.
• Contextual chunks: give each chunk back its context before indexing. The cheap option prepends the document title and section. The richer option is an LLM-written note situating the chunk in its document; Anthropic calls this contextual retrieval and applies it to both the embeddings and the BM25 index (source above). The richer option costs an extra model call per chunk at ingest.
• Structure-aware splitting: respect headings, tables and code blocks.
• Diversity (MMR, maximal marginal relevanceMaximal marginal relevance (MMR)A selection method that balances relevance against redundancy, so the top-k results are not near-duplicates.Open the glossary entryClose): trade relevance against redundancy so the top-k is not five copies of the same paragraph.

Context assembly#

The last step before generation:

• De-duplicate, order by relevance, and cap by a token budget.
• Put the strongest chunks at the start or end, because models use the middle of long contexts less reliably (Liu et al., 2023, “Lost in the Middle”).
• Include source ids so the model can cite.
• Optionally compress by extracting only the relevant sentences. The risk is dropping a sentence the answer needed.

Debugging playbook#

Work from the symptom back through the pipeline. Ask one question at a time.

Diagram: A decision tree for a bad answer: first check the passage is indexed, then that it was retrieved, then that it survived reranking and...

Test to separate retrieval from generation: paste the gold passage into the prompt by hand. A correct answer means retrieval (or ingestion) failed. A wrong answer means generation or prompt.

Log the retrieved chunk ids for every request. Without them this tree cannot be walked after the fact.

Real-world application#

Teams usually add these in order of cost: filters and chunk fixes first (no extra runtime calls), then hybrid search, then reranking, and query transforms last because they add an LLM call to every request. Each step is justified by a failure you have measured on a labelled set (see RAG evaluation in depth).

Interview perspective#

Provenance: common industry pattern; no company-specific claim.

• “Dense for meaning, BM25 for exact tokens, fuse with RRF, rerank the shortlist with a cross-encoder.”
• “I log retrieved chunk ids per request so every bad answer is debuggable.”
• “Permissions are filters in the retriever, not instructions in the prompt.”

Practice questions#

Provenance: generated practice.

• Why does RRF avoid score normalisation?
• When would multi-query hurt?
• Pre-filter versus post-filter: what is the trade-off?
• Design the test that tells a retrieval failure from a generation failure.

Activity#

Do, notice, apply.

1. Do. Implement RRF over two ranked lists (a few lines of code, or by hand with k = 60). Use lists where one document is first in one list and missing from the other.
2. Notice. Build a case where RRF beats both input lists. What property of the two lists made that possible? What happens to a document that appears in only one list? Describe each document by its rank in each list, and by the rank of a known relevant document.
3. Apply. Take the debugging tree and a bad answer from any RAG tool you use. Write which question in the tree you would ask first, and what log you would need to answer it.

Homework#

For a failing question set you invent (five questions, each with a different cause), write the first fix you would try for each, the metric you would watch, and its latency or cost price.

Quick revision#

• Quality is capped by retrieval. The right chunk must reach the prompt.
• Hybrid = dense + BM25, fused by RRF (k = 60 default, rank-based).
• Rerank a wide shortlist with a cross-encoder: more precision, more latency.
• Query transforms fix the query but add a call and can drift. Gate them.
• Filter by metadata and permissions before or during search.
• Log retrieved chunk ids. Walk the debugging tree from the symptom back.

Check yourself#

• Checkpoint (answer aloud, no notes): Why does RRF avoid score normalisation, and when would you not use it?
• Re-test (nearby variant): Exact SKU searches fail, but paraphrased questions work. Which technique do you reach for and which metric tells you it worked?
• Misconception to avoid: “Reranking fixes recall.” It can only reorder what retrieval already returned.

Related#

• The core RAG pipeline
• All RAG types
• RAG evaluation in depth