Fine-tuning methods: SFT, LoRA, QLoRA and DPO

Story

Hide story

The siding has room to change Tobias once tonight, and only once. A full retraining would keep the train here past sunrise.

“Something small,” Marta says, tapping the slip. “Something we can check by morning.” Merrow, who has seen small changes go wrong, nods slowly. Tobias asks whether it will hurt.

Which kind of change fits the time and the space we have?

Concept#

Fine-tuning vs RAG vs prompting covers whether to fine-tune. This lesson covers how. A modern model passes through several training stages, and each uses different data to teach something different.

1. PretrainingPretrainingThe first, large training stage: next-token prediction over a huge text corpus. Most language ability and world knowledge come from it.Open the glossary entryClose. Next-token prediction over a huge text corpus. This is where most of the language ability and world knowledge come from.
2. SFTSFTShort for supervised fine-tuning: fine-tuning on pairs of a prompt and an ideal response.Open the glossary entryClose (supervised fine-tuning, also called instruction tuning). Train on (prompt, ideal response) pairs so the model follows instructions in a given format, style or task routine.
3. Preference tuning. Teach “better versus worse” from comparisons of two responses.
   • RLHFRLHFReinforcement learning from human feedback: train a reward model on preferences, then optimise the model against it with reinforcement learning.Open the glossary entryClose trains a reward model on human preference pairs, then optimises the model with reinforcement learning (often PPO), as in InstructGPT (Ouyang et al., 2022). A penalty on the KL divergence from the SFT model keeps it from drifting into odd outputs that merely score well.
   • DPODPODirect preference optimization: trains directly on preference pairs, without a separate reward model or reinforcement-learning loop. It still uses a frozen reference copy of the model.Open the glossary entryClose (Rafailov et al., 2023) reparameterises the problem so the model is trained directly on preference pairs with a classification-style loss. There is no separate reward model and no reinforcement-learning loop, which makes it simpler and often more stable, though it still needs a frozen reference copy of the model.
4. Distillation. Train a small model on a large model’s outputs to keep a narrow task’s quality at lower cost and latency.

Why it matters#

Fine-tuning questions come in two forms: mechanism (“what is LoRA actually doing?”) and judgement (“what data, how much, how do you know it worked, and what did it break?”). This lesson covers the mechanism and the choice of method: knowing the stages tells you which data to collect, and knowing the arithmetic tells you what it costs. The judgement half is in Fine-tuning data, risks and shipping.

Mental model#

Full fine-tuning rewrites the whole textbook. LoRALoRALow-rank adaptation: freeze the base weights and train a small low-rank update, a tiny fraction of the parameters. Adapters are small, swappable and mergeable.Open the glossary entryClose leaves the textbook alone and clips a thin, removable overlay onto each page: the original weights stay frozen and you learn a small correction. Because only the overlay is trained, you can store many overlays and swap them.

Where the analogy breaks: an overlay is not literally additive text. The correction is a matrix added to a weight matrix, and it changes how every input passes through that layer.

SFT and preference data#

SFT data are pairs: a prompt and the response you want. The model learns to produce that response given that prompt. Preference data are triples: a prompt, a preferred response and a rejected one.

• Use SFT to teach format, tone, task routines and tool-use habits.
• Use preference tuning to shape judgement between acceptable answers, such as helpfulness versus verbosity or safe versus unsafe, where there is no single correct output to imitate.
• DPO’s loss compares how much more likely the tuned model makes the preferred response than the rejected one, relative to the frozen reference model. A strength parameter (beta) controls how far the model may move from the reference.

LoRA#

Low-rank adaptation (Hu et al., 2021) freezes a pretrained weight matrix W of shape d × k and learns a correction ΔW = B·A, where B is d × r, A is r × k, and the rank r is much smaller than d and k. Freezing W is what keeps this cheap: no gradients or optimiser state are kept for the big matrix, and only A and B are updated. The layer computes Wx + (α/r)·B·A·x, where α is a scaling constant. A starts random and B starts at zero, so training begins from exactly the original model.

Diagram: The input flows through the frozen weight matrix and, in parallel, through the small trainable pair A (down to rank r) and B (back up to d).

Worked example: counting trainable parameters#

A 4096 × 4096 layer has 4096 × 4096 = 16,777,216 weights, about 16.8 million. A rank-8 adapter has A of 8 × 4096 = 32,768 and B of 4096 × 8 = 32,768 weights, so 65,536 in total. That is 65,536 / 16,777,216 = 0.39% of the layer.

The fraction doubles when the rank doubles. Choosing which matrices get adapters (for example the attention projections) is a separate design choice, and a model has many such layers, so the whole-model fraction depends on that choice.

In the original paper, compared with fully fine-tuning GPT-3 175B with Adam, LoRA reported about 10,000 times fewer trainable parameters and about 3 times lower GPU memory, with no added inference latency because the adapter can be merged into W.

Practical consequences:

• Small files. An adapter is a small fraction of the size of a full copy of the model.
• Swappable. One base model can serve many adapters, for example one per task or tenant.
• Mergeable. Fold B·A into W for serving with no extra latency, or keep the adapter separate to stay swappable.
• Limits. Low rank restricts how much the model can change. It suits style, format and narrow domain behaviour. A very large shift may need a higher rank or full fine-tuning.

QLoRA#

QLoRAQLoRALoRA on top of a base model held in 4-bit precision, which cuts memory a lot at the cost of slower steps. Check quality on your task.Open the glossary entryClose (Dettmers et al., 2023) keeps the frozen base model in 4-bit precision (a data type called NormalFloat 4, or NF4), runs gradients through it, and trains LoRA adapters in higher precision. It also uses double quantisation (quantising the quantisation constants) and paged optimisers to handle memory spikes. The authors report that this fine-tunes a 65B-parameter model on a single 48 GB GPU while preserving 16-bit fine-tuning task performance in their experiments.

Memory arithmetic for weights only: a 7-billion-parameter model at 16 bits (2 bytes each) needs 7 × 10⁹ × 2 = 14 GB. At 4 bits (0.5 bytes) it needs about 3.5 GB. Activations, optimiser state, adapters and quantisation constants come on top, so real usage is higher.

The trade-off is speed and fidelity: dequantising weights during training typically makes steps slower than 16-bit LoRA, and the quality match is a reported result in specific settings. Check it on your own task before relying on it.

Choosing a method#

Picking a method is half the job. The next lesson, Fine-tuning data, risks and shipping, covers the data you train on, what a tune can break, and how to prove it beats the baseline before you ship it.

Real-world application#

A team wants a support assistant to answer in a fixed house format on one 7-billion-parameter base model. They train a LoRA adapter on a few thousand prompt and ideal-response pairs rather than retraining the model, because only format and tone change. If the format were a large behaviour shift, they would revisit the rank or full fine-tuning.

Interview perspective#

Provenance: common industry pattern; no company-specific claim.

Expect “explain LoRA” and “RLHF versus DPO”. Weak answers claim fine-tuning teaches new facts reliably, or cannot say what LoRA freezes and what it trains. The questions on data, regressions and shipping are in Fine-tuning data, risks and shipping.

Practice questions#

Provenance: generated practice.

• Basic: What does LoRA freeze, and what does it train?
• Intermediate: RLHF versus DPO: what is removed, and what is the trade-off?
• Advanced: Your LoRA run on a 13-billion-parameter model needs more memory than your GPU has. What do you change, and what do you accept in return?
• Product: A customer wants a house style on a shared base model. Do you use full fine-tuning or an adapter, and why?

Activity#

Do, notice, apply.

1. Do. Compute the trainable fraction for ranks 4, 8 and 16 on a 4096 × 4096 layer, then repeat for an 8192 × 8192 layer.
2. Notice. What happens to the fraction when the layer gets bigger at the same rank?
3. Apply. Argue which rank you would start with, and the experiment that would tell you whether to raise it.

Homework#

Write a one-page method decision note for a model you could plausibly tune: the hardware you have, the size of the behaviour change you need, the weight-only memory arithmetic for 16-bit and 4-bit, your choice of full fine-tuning, LoRA or QLoRA, and the trade-off you accept.

Quick revision#

• Pretraining gives knowledge; SFT teaches behaviour; RLHF and DPO teach preference; distillation shrinks.
• LoRA: frozen W plus a low-rank B·A, about 2·d·r trainable weights for a d × d layer, swappable and mergeable.
• QLoRA: 4-bit frozen base plus higher-precision adapters, lowest memory, slower steps.
• Choose by hardware and size of change: full fine-tune, LoRA or QLoRA.

Check yourself#

• Checkpoint (answer aloud, no notes): Explain LoRA to a backend engineer, including why it is cheap and what it cannot do.
• Re-test (nearby variant): Explain the same idea for a 70B model on one GPU: what changes when the base is held in 4-bit?
• Misconception to avoid: “fine-tune on our docs so it knows them.”

Related#

• Fine-tuning vs RAG vs prompting
• Fine-tuning data, risks and shipping
• LLM inference and serving