RAG architecture and chunking

Story

Hide story

Merrow has cut the timetable into strips so each one fits a card drawer. You pull one at random. It reads: “…and so it departs at 22:10, calling at all stops.”

Departs from where? Which train? Merrow looks at it, stamps it FILED, and slides it into a drawer.

Later a passenger asks about the 22:10. Will anyone ever find this strip, and if they do, will it help?

Concept#

Retrieval-augmented generation (RAG)RAGShort for retrieval-augmented generation: putting retrieved content in the prompt so the answer is grounded in it.Open the glossary entryClose retrieves relevant pieces of text from an external store and puts them into the prompt, so the model answers from real, current or private documents instead of from its training data alone. The model’s weights do not change.

The pieces are called chunksChunkA piece of a document that is embedded and retrieved as one unit. Its size and boundaries strongly affect retrieval quality.Open the glossary entryClose. How you cut documents into chunks decides what the retriever can find later.

Why it matters#

A model has a training cutoff, no access to your private documents, and no way to cite where a claim came from. RAG addresses all three. It reduces hallucination, because the model has concrete text to answer from, but it does not remove it (see RAG evaluation and hallucination).

Chunking matters because retrieval can only return what ingestion produced. If the strip that holds the answer was cut badly, no later stage can reliably recover it. In practice, many “the model got it wrong” reports trace back to this step.

Mental model#

RAG is an open-book exam. You do not expect the model to have memorised everything. Right before it answers, you hand it the few pages that matter.

Chunking is how you decide what a “page” is. Too small and a page says “and so it departs” with no train named. Too big and the page holds a hundred topics, and its single summary vector matches none of them well.

The pipeline in two phases#

RAG has an ingestIngestionThe offline part of a RAG system: parse, chunk, embed and index documents. Safe ingestion is idempotent, incremental and records model and chunker versions.Open the glossary entryClose phase that runs ahead of time and a query phase that runs on every question.

Diagram: Ingest turns documents into embedded chunks in a vector store.

Ingest, step by step:

1. Parse and clean. Turn PDFs, HTML and wikis into clean text. Garbage here is garbage everywhere.
2. Chunk. Split the text into pieces (the rest of this lesson).
3. Embed. Turn each chunk into a vector (see Embeddings and similarity search).
4. Store. Save vectors plus metadata in a vector database.

Query, step by step: embed the question with the same model, fetch the top-k nearest chunks, optionally rerank them, build a prompt from the question and the chunks, and generate.

Because ingest is separate from the model, updating knowledge means re-ingesting the changed document. No retraining is involved. That is the main reason RAG suits facts that change.

Chunking trade-offs#

You cannot embed a 50-page document as one vector. One vector is one summary of everything in it, so it dilutes relevance and loses the specific detail a question asks about.

Common splitting strategies:

• Fixed size (by tokens or characters): simplest, ignores meaning.
• Sentence or paragraph based: respects natural units.
• Semantic: split where the topic changes. Better boundaries, but costs extra computation at ingest.
• Structure-aware: follow headings, sections, lists and tables. Usually the best default for documents that have structure.

There is no universal best size. Illustrative starting points that teams try are a few hundred tokens with 10 to 20 percent overlap. Treat them as a first experiment, not an answer (the last section shows how to decide).

Text summary

This tool cuts one short, made-up passage of 83 words (a lending guide for an imaginary library) into chunks of words. You set the chunk size and the overlap, and overlap is always smaller than the size. The question is “How many times can a book be renewed?”, and the answer is one sentence: “Books can be renewed twice, and each renewal adds fourteen days to the loan.” The tool lists the chunks and marks each one that holds the whole answer sentence, or only part of it. At a chunk size of 20 words with no overlap, the cut falls inside the answer sentence, so no chunk holds all of it and the tool warns you. With an overlap of 8 words, the second chunk holds the whole sentence. More overlap means more words stored, because the repeated words are kept in both chunks. Chunks are counted in words here; real systems usually count tokens or characters.

The interactive version needs JavaScript. The text summary above says what it shows.

Chunk by structure, not by character count#

Splitting every N characters can cut a sentence, a clause or a table in half. Respect the document’s own boundaries instead.

Headings and sections. Split on headings, then on paragraphs, and only fall back to a size limit inside a long section. Store the heading path (for example “Refunds > Digital goods”) with each chunk.

Tables are the hard case. If you split a table row by row, a row such as “Gold | 14 days | 5%” has lost its column names. A reader, and an embedding, cannot tell what 14 or 5% mean. Options:

• Keep a small table as one chunk.
• For a large table, repeat the caption and header row in every chunk.
• Or turn each row into a sentence (“Gold plan: refund window 14 days, fee 5%”) before embedding.

Parsing quality comes first: a multi-column PDF read in the wrong order produces nonsense no chunker can fix. See Document AI for layout-aware parsing.

Give chunks their context back#

An isolated chunk often says “it”, “the company” or “this quarter” without naming what. Three common fixes:

• Cheap prefix. Before embedding, prepend the document title and section heading to each chunk. It costs nothing at query time and often fixes “which document is this about?” confusion.
• LLM-written context note. A model reads the whole document and writes a short note situating each chunk in it (for example, which filing and which quarter a revenue sentence belongs to). Anthropic calls this contextual retrievalContextual retrievalA chunking technique where a model writes a short note placing each chunk in its document, and the note is added to the chunk before embedding and BM25 indexing. It costs one extra model call per chunk at ingest.Open the glossary entryClose: the note is prepended to the chunk and used for both the embeddings and the BM25 index (source above; read it for their setup and results). The cost is an extra model call per chunk at ingest.
• Small-to-big (parent-child)Parent-child retrievalMatching on small chunks and then returning their larger parent section. It gives precise matching with more context, at the cost of bigger prompts.Open the glossary entryClose. Embed small chunks for precise matching, then hand the model the larger parent section. You pay with larger prompts and need to de-duplicate parents that several children point to.

Metadata and permissions#

Each chunk should carry fields you can filter on, saved at ingest time:

• source (document id or URL), so you can cite it;
• section or heading path;
• date or version, so you can prefer current text and drop superseded text;
• language, document type;
• access control (tenant, group, or user permissions).

Access control is the field people forget, and it is the one with legal consequences. If a user may not read a document, its chunks must never reach the prompt for that user.

Enforce this inside the retrieval query (a filter or a per-tenant namespace). Do not retrieve first and filter afterwards:

• Post-filteringPost-filteringApplying a metadata filter after the nearest-neighbour search. Filtered-out candidates leave fewer than k results. The alternative is filtering inside the search; see metadata filter.Open the glossary entryClose a top-k can leave you with zero results, because the best matches were the ones removed.
• Scores, summaries, logs and caches built from unfiltered results can leak the existence or content of restricted documents.
• Asking the model to “not mention” restricted text is not a control. The text is already in the prompt.

Also scope any cache by identity, so one user’s cached answer is never served to another.

Choosing chunk size with evidence#

Folklore (“512 is standard”) is how teams end up with a bad default for years. The corpus and the questions decide.

1. Build a small labelled set. Collect realistic questions and mark which passage answers each. Start with a few dozen (an illustrative size; see RAG evaluation in depth for building a proper one).
2. Change one thing. Index the corpus at two or three chunk sizes, with the same embedder and the same overlap.
3. Measure retrieval. For each question, check whether a chunk containing the answer appears in the top-k. The fraction of questions where it does is recall@k. Also look at where in the list it appears.
4. Check the prompt cost. Bigger chunks mean more tokens per query. A small recall gain may not justify doubling the context.
5. Check the end-to-end answer on a handful of examples. Retrieval can improve while the answer worsens if large chunks bring in noise.

Different content often wants different settings: short FAQ entries need almost no splitting, while long contracts need clause-aware chunks with their section titles attached.

Real-world application#

This is the architecture behind internal-document Q&A and support-over-knowledge-base products. In production, the ingest side is a pipeline with its own schedule, monitoring and re-index plan, because changing the chunker or embedder means re-processing the whole corpus.

Interview perspective#

Provenance: common industry pattern; no company-specific claim.

Expect “walk me through a RAG pipeline”, “how do you choose chunk size”, “how do you handle tables”, and “what happens if retrieval returns nothing relevant”. A strong answer names the trade-off, then says how you would measure it. A common mistake is conflating RAG with fine-tuning, or answering chunk size with a number and no method.

Practice questions#

Provenance: generated practice.

• Basic: What are the two phases of a RAG pipeline, and which one runs per question?
• Intermediate: Why does chunk overlap help, and what does it cost?
• Advanced: Your corpus is long legal contracts plus short FAQ entries. Would you use one chunking setup for both?
• Product company: Retrieval returns three moderately relevant chunks and none answers the question. What should the system do?

Activity#

Do, notice, apply.

1. Do. Take a three-paragraph document you have (a policy, a README, a manual page). Cut it two ways: into single sentences, and into one chunk per paragraph. Write three questions a reader might ask about it.
2. Notice. For each question, which cut keeps the information needed to answer it? Which cut makes a chunk that could not be understood alone (a “…and so it departs” strip)?
3. Apply. Now take a document that contains a table. Decide how you would chunk it so a row still makes sense, and write the chunk text for one row.

Homework#

On paper, design the ingestion pipeline for a company’s internal HR policy documents. State the inputs and parsing choices, the chunking approach and why, the metadata fields (including who may read what), and how you would test your chunk size.

Quick revision#

• RAG retrieves chunks and puts them in the prompt. Weights do not change.
• Ingest runs ahead of time. Query runs on every question.
• Small chunks are precise but context-poor. Large chunks are context-rich but blurry. Overlap protects boundaries at some cost.
• Split by structure, keep table headers with rows, and restore context with a prefix or a parent.
• Store source, section, date and permission metadata. Enforce permissions in the query.
• Choose chunk size by measuring recall@k on a labelled set.

Check yourself#

• Checkpoint (answer aloud, no notes): Design chunking and retrieval for 10,000 PDFs with tables and headings. Justify chunk size, overlap and metadata, and say how you would choose empirically.
• Re-test (nearby variant): Answer quality dropped after raising chunk size from 256 to 1024. Give two mechanisms and say how you would tell whether retrieval or generation is at fault.
• Misconception to avoid: picking a chunk size by folklore, embedding a whole document as one vector, or ignoring tables.

Related#

• Vector databases
• The core RAG pipeline
• RAG evaluation and hallucination