Lesson deck
Fine-tuning methods: SFT, LoRA, QLoRA and DPO flashcards
Reviewing needs JavaScript. You can read every card in the list below.
Loading your cards...
Back to the lesson5 cards in this deck
Read all 5 cards as a list
- Front
- Explain LoRA in two sentences.
- Back
- Freeze the weights W and learn a low-rank update B x A (rank r much smaller than d), a tiny fraction of the parameters. Adapters are small, swappable and mergeable into W.
- Front
- RLHF vs DPO?
- Back
- RLHF trains a reward model then optimises with RL. DPO trains directly on preference pairs with a classification-style loss: no separate reward model or RL loop, though it still uses a frozen reference model.
- Front
- How many trainable weights does a rank-r LoRA adapter add to a d x d layer?
- Back
- 2 x d x r (A is r x d, B is d x r). For d = 4096, r = 8: 65,536, about 0.39% of the layer's 16.8 million weights.
- Front
- What does QLoRA add to LoRA?
- Back
- The frozen base is held in 4-bit (NF4) while adapters train in higher precision, with double quantisation and paged optimisers. It cuts memory a lot, with slower steps; check quality on your task.
- Front
- Which data does SFT use, and which does preference tuning use?
- Back
- SFT: prompt and ideal-response pairs. Preference tuning: a prompt with a preferred and a rejected response.