Fine-tuning vs RAG vs prompting

Story

Hide story

In the lamp-light of the siding, Marta lays the week’s complaint slips in rows. Tobias keeps stumbling over the same kind of announcement: delays. He reads them out as though they were good news.

Marta writes three options on the back of a slip. Send him back to the training school. Have Merrow hand him better pages. Give him a clearer note before he steps up to the microphone.

“Pick one,” she says. What would you want to know first?

Concept#

When a model does not do what you need, you have three levers, and each changes a different thing.

1. Prompting changes the instructions and examples in the input. No training, and you can iterate in minutes.
2. RAGRAGShort for retrieval-augmented generation: putting retrieved content in the prompt so the answer is grounded in it.Open the glossary entryClose changes what information the model sees at answer time, by retrieving documents into the prompt. The model’s weights do not change.
3. Fine-tuningFine-tuningUpdating a model's weights on labelled examples to teach a consistent behaviour, style or format. It is a poor way to add or update facts.Open the glossary entryClose changes the model’s weights by training on labelled examples. It teaches a consistent pattern of behaviour: a format, a tone, a task routine.

The levers answer different questions. Prompting asks “did I explain the task clearly?” RAG asks “does the model have the facts?” Fine-tuning asks “can the model do this reliably without long instructions?”

Why it matters#

“Let us fine-tune it so it knows our product” is a common proposal and often the wrong one. A team can spend weeks and a training budget, then find that answers are still wrong, still uncited, and out of date the next time prices change. Choosing the lever by the kind of problem saves that cost.

Mental model#

RAG is handing someone the right reference book before they answer. Fine-tuning is coaching how they write and work: their house style, their checklist. After coaching, they still do not hold your company’s current price list in their head, and if you ask for a fact they never saw, they may still produce a fluent guess.

The analogy breaks in one place: coaching a model can also nudge it toward saying “I don’t know” in the right situations, which helps (see below).

The decision#

Ask what the problem actually is, then follow the cheapest lever that addresses it.

Diagram: The decision starts from the type of problem.

Two further points about order:

• Cheapest first is about iteration speed and risk, not a strict ladder. Prompting and RAG are both reversible in minutes. A fine-tune takes a data pipeline, a training run and an evaluation, so it comes after you have a baseline.
• The levers combine. A common production system is a fine-tuned or well-prompted model that follows a strict format, fed by RAG for the facts. They are not alternatives you pick once.

What fine-tuning does poorly#

• Changing facts. Knowledge baked into weights goes stale. Updating one price means another training run, while a RAG index can update the document. Facts are also hard to remove or correct later.
• Citations. A fine-tuned model has no source document to point at. RAG gives you the retrieved passage, which you can cite and audit.
• Per-user access control. Weights are shared by every user. A document-level permission cannot be applied to knowledge inside the weights, but a retrieval step can filter by user. Training on restricted data risks leaking it to everyone.
• Teaching new facts reliably. In a controlled closed-book question-answering study, Gekhman et al. (EMNLP 2024) found that examples introducing new knowledge were learned more slowly than examples the model already knew, and that as the model absorbed them its tendency to hallucinate rose. That is one setting, so treat it as a warning about the mechanism, not a law.

Does fine-tuning fix hallucination?#

Not by itself, and training on new facts may increase it, per the study above. But the answer has a nuance. Fine-tuning on examples that include correct refusals (for unanswerable questions, “I can’t find that”) can improve abstention behaviour: R-Tuning (Zhang et al., NAACL 2024) builds refusal examples from questions that fall outside what the model already knows, and reports more refusals of unknown questions and better answers to known ones. A model can also be tuned to answer in a fixed grounded format, for example always quoting a passage id from the supplied context. Those are behaviours. The reliable way to supply correct facts is still to retrieve them.

What fine-tuning does well#

• Field accuracy, token cost and consistency in a structured task at high volume, where long instructions in every prompt cost tokens and are still followed unevenly. Schema validity is a separate matter: it comes from constrained decodingConstrained decodingRestricting which tokens the model may produce at each step so the output must fit a schema. The shape conforms; the content can still be wrong.Open the glossary entryClose or a validator (see Reasoning and structured outputs), not from training.
• A house style or tone that prompts reproduce unevenly.
• Domain-specific jargon or a routine such as triage categories, where you have many labelled examples.
• Tool-use habits for a specific tool set.
• DistillationDistillationTraining a small model on a large model's outputs to keep a narrow task's quality at lower cost and latency.Open the glossary entryClose: training a small model on a large model’s outputs to cut cost and latency for a narrow task. Details are in Fine-tuning methods.

What must exist before you start#

Fine-tuning is worth starting when all of these are true.

1. Labelled examples. Often hundreds to thousands of good input and output pairs (the right number depends on the task; start small and measure). Noisy labels train noisy behaviour.
2. A baseline to beat. The best prompt, with RAG if facts are involved, measured on an eval set. A fine-tune must beat this strong baseline, not a first-draft prompt.
3. An evaluation. Task metrics, a regression check on general abilities, and a held-out set. See Evaluating LLM systems.
4. A reason that prompting cannot meet. Examples: consistent behaviour across millions of calls, a latency or cost target that needs a smaller model, or a prompt that has grown too long to maintain.
5. A maintenance plan. Base models get replaced. A fine-tune is tied to one base model, so you need to be able to rerun the training on new data and a new base.

Provider support for fine-tuning differs and changes, including which models can be tuned. Check the current documentation of the provider before planning around it.

Real-world application#

Using fine-tuning to raise field accuracy and cut instruction tokens in a high-volume extraction pipeline is a legitimate use (schema validity itself comes from constrained decoding). Using it to “teach the model our pricing” is the mistake: prices change, the model cannot cite where a number came from, and a wrong price still comes out fluently. The same product can use both, with a tuned model for format and RAG for the price table.

Interview perspective#

Provenance: common industry pattern; no company-specific claim.

Expect “when would you fine-tune instead of using RAG” and “can fine-tuning fix hallucination”. A strong answer separates knowledge from behaviour, names the baseline and eval, and mentions staleness, citations and access control. A weak answer treats fine-tuning as a universal upgrade.

Practice questions#

Provenance: generated practice.

• Basic: Which lever is fastest to try first, and why?
• Intermediate: Why does fine-tuning on a wiki not give you a reliable question-answering bot?
• Advanced: Give two cases where fine-tuning is genuinely the right choice and the evidence you would need.
• Product: A team wants to fine-tune a model on last quarter’s sales data so it can answer sales questions. What is wrong with this plan, and what would you propose?

Activity#

Do, notice, apply.

1. Do. Assign each requirement to prompting, RAG, fine-tuning or a combination: “answer from our latest pricing sheet”, “always respond in our brand’s tone”, “extract data in our exact schema from 2 million emails a day”, “answer only what the user’s role may see”.
2. Notice. Which assignments changed once you added volume, freshness or permissions to the requirement?
3. Apply. For any requirement you gave to fine-tuning, write the baseline you would beat first.

Homework#

Write a one-page decision memo for a customer-facing feature of your choice. For each lever, give what it would change, the cost, the risk and the evidence that would make you choose it.

Quick revision#

• Prompting changes instructions. RAG changes the information in the prompt. Fine-tuning changes weights.
• Facts go to RAG: current, citable and permissioned.
• Fine-tuning shines for consistent format, style and behaviour at volume, and for distillation.
• Fine-tuning alone does not cure hallucination; it may help abstention and format.
• Before tuning: labelled data, a strong baseline, an eval, a reason prompting cannot meet.

Check yourself#

• Checkpoint (answer aloud, no notes): A team wants to “fine-tune on our wiki so the bot knows it”. Give the decision order and why this usually disappoints.
• Re-test (nearby variant): When is fine-tuning right here? Give two concrete cases and the data and eval you would need.
• Misconception to avoid: using fine-tuning to inject frequently changing knowledge.

Related#

• Prompting fundamentals
• RAG architecture and chunking
• Fine-tuning methods