Evaluation and model adaptation quiz

Choose one answer.

• An LLM judge with a rubric, because format quality is subjective and a model reads structure more flexibly than a parser.
• A deterministic schema check in CI, because validity is a yes or no property that code decides exactly and cheaply.
• Human review of a sample, because a person is better placed to confirm that the required fields hold the right content.
• Online A/B signals such as retries, because users will reveal malformed output faster than any offline test does.

Choose one answer.

• The metrics came from easy-to-compute ideas, not from the failure categories real traces show, so the main failures stay unmeasured.
• Similarity metrics suit language models poorly, so the numbers the team reported were invalid from the start.
• The score should have come from human reviewers, because an automatic metric tends to drift away from user complaints.
• The dataset was too large, so averaging over many cases diluted the few complaints that real users were raising.

Choose one answer.

• Eighty cases is too few for any score, so the team should double the set with more thumbs-up chats until the interval narrows.
• A perfect score means the judge is too lenient, so the team should swap in a stricter judge model and keep the same set.
• The set leaves out the failures, so the team should sample logged production traces, sort the failures into categories and add them.
• Thumbs-up chats use the wrong format, so the team should rewrite them in a more formal tone before running the comparison.

Choose one answer.

• 0.30, found by subtracting the chance agreement of 0.50 from the observed agreement of 0.80 and stopping there.
• 0.80, because kappa equals the share of items where the judge and the human gave the same label.
• 0.50, because kappa is bounded by the chance agreement that two labellers would reach without looking at the items.
• 0.60, found by (0.80 - 0.50) / (1 - 0.50), the improvement over chance as a share of the possible improvement.

Choose one answer.

• Set the judge temperature to 0, so the same pair produces the same verdict on repeated runs.
• Judge each pair twice with the order swapped, and count a win only when both orders agree.
• Switch to a judge from the same model family as the system, because shared training removes disagreements about style.
• Ask the judge for a longer written justification before it chooses, because reasoning at length makes the choice neutral.

Choose one answer.

• Accept the judge, since 90% agreement is above the level that strong judges reached against humans in published chat studies.
• Retire the judge, since agreement below 100% suggests it should not be used to score outputs at scale.
• Compute kappa and check agreement on the rare fail class, because chance agreement is high with such a skewed label balance.
• Rebalance the labelled sample by deleting passes until the classes match, then report the old 90% figure as before.

Choose one answer.

• Ship it, because any positive difference means the new prompt is better, and the old prompt will stay available as a rollback.
• Reject it, because a gain below 5 points is too small to matter for any product, whatever the noise level.
• Treat the evidence as inconclusive and compare the same cases paired, with a bootstrap interval or more cases.
• Double the number of judge calls per case, because repeated judging removes the sampling noise in the dataset itself.

Choose one answer.

• The edits fitted the held-out cases, so that set now acts as a second dev set, and a fresh held-out set is needed.
• The held-out set was too small to be useful, so the team should have used the dev set to gate releases.
• Production quality is measured on a different scale, so offline scores cannot be compared with online outcomes directly.
• The judge has drifted since the set was built, so rerunning the same judge a second time will correct the score.

Choose one answer.

• A confirmed regression for that customer, since a 20 point drop is larger than any gain seen on the other slices.
• Noise dominates at this size, so keep the slice under watch, add cases for it, and compare paired before acting.
• A reason to drop the slice from reporting, since small slices cannot say anything useful about a release decision.
• A judge fault, since a slice that falls while the others stay flat points to scoring drift on those inputs.

Choose one answer.

• Pick A, because a public benchmark is run by independent parties and so is more objective than your own labels.
• Average the two rankings with equal weight, because both are evidence and neither one should dominate the final decision.
• Rerun the public benchmark with a different prompt until A and B swap, then treat the disagreement as settled.
• Check your eval's labels and sample for errors, then prefer it, since it measures your tasks and avoids contamination.

Choose one answer.

• The internal set is mislabelled, so the model's real accuracy is the 99% shown and the internal items need fixing.
• Judge position bias on the internal set, since the benchmark used fixed answers and the internal set used pairwise judging.
• Benchmark saturation: top models cluster near the ceiling, so the 99% shows the model is as good as any other.
• Benchmark contamination: the test items or close copies were in the training data, so the 99% reflects memory.

Choose one answer.

• Fine-tune on the latest policy text, because training makes the model reliable at reproducing the wording it was shown.
• Retrieve the current policy at answer time with RAG, so an update to the document is an update to the bot.
• Paste the current policy into the system prompt at launch, because the text is short enough to fit.
• Distil a small model from a larger one, because a smaller model memorises a fixed policy more tightly.

Choose one answer.

• Move the field definitions into a RAG index so the model retrieves the extraction rules for each email before extracting.
• Stop at 91%, because fine-tuning becomes justified once the prompt gets under 80% correct on a task like this one.
• Fine-tune on labelled examples to raise field accuracy and cut prompt tokens, then compare with the prompted baseline.
• Add a judge model that scores every output for schema validity, because a model reads structure better than a parser.

Choose one answer.

• Add explicit length and tone instructions plus a few worked examples, then measure on a small test set.
• Launch a fine-tuning run on logged summaries, because inconsistency of style is what training on examples solves.
• Build a RAG index of past summaries, because retrieving earlier output will make the model copy its tone.
• Switch to a larger model before changing anything, because size alone enforces consistent length and tone.

Choose one answer.

• Retrain the tuned model each month on the new fares, so the weights hold the current prices and the format together.
• Drop the tuned model and put the format rules back into a long prompt, since weights cannot be used alongside fresh facts.
• Add a line to the prompt telling the tuned model to remember that fares change monthly and to answer with extra care.
• Keep the tuned model for the format and supply current fares through retrieval at answer time, so a fare edit needs no training.

Choose one answer.

• A wiki is a small corpus, so the model would not pick up enough from it to answer any question well.
• Weights are shared by every user, so manager-only knowledge cannot be filtered per person, and answers have no source to cite.
• Fine-tuning erases the model's general knowledge outside the wiki, so it could no longer answer everyday questions about other subjects.
• A fine-tuned model declines to answer from documents it saw in training, so HR questions would go unanswered.

Choose one answer.

• It fixes hallucination only for the topics covered in training data, and for those topics the answers become verifiable.
• It fixes hallucination once enough examples are used, because the model memorises the true answers and recalls them correctly.
• It has little effect on hallucination, because weights set what the model knows but not how it behaves when unsure.
• It can improve abstention and format when trained on refusal and grounded examples, but it does not supply correct facts.

Choose one answer.

• New facts are learned slowly, and once absorbed they push the model toward confident guesses on facts it never learned.
• The training run was too short, since longer training on new facts removes the extra wrong answers in the same study.
• The model forgot its older knowledge, which the study showed as a drop in accuracy on the facts it already knew.
• The retrieval index was out of date, because fine-tuned models read from an index at answer time to find facts.

Choose one answer.

• A user survey asking whether people like the tuned model, because offline numbers do not predict satisfaction.
• A second fine-tuning run on twice the data, because only repeated training shows the gain is real.
• A larger model to replace the base model, because gains from tuning shrink when the base grows.
• A comparison against the base model with the best prompt, plus RAG if facts matter, on a held-out set.

Choose one answer.

• Competitors announced tuned models, and the team wants an equivalent in its roadmap this quarter.
• The assistant must know the latest prices, which change weekly, and training seems more robust than a lookup.
• Prompting reaches 92% correct field values, the remaining errors cost money at volume, and 5,000 examples exist.
• The prompt is long and unloved, so removing it by training feels cleaner, whatever the quality result may turn out to be.

Choose one answer.

• Preference pairs of preferred and rejected responses, used for RLHF or DPO to shift the model toward the better answer.
• Plain unlabelled text, continued with next-token prediction, so the model absorbs concise writing from the extra corpus.
• A set of prompts with a single ideal response each, trained by SFT to reproduce that wording exactly.
• A larger model's outputs on your prompts, used to distil a smaller model into the same concise style.

Choose one answer.

• The preference data itself, because DPO infers which response is better from the model's own likelihoods without human comparisons.
• The frozen reference model, because the tuned model is compared only with itself during training.
• The separate reward model and the reinforcement-learning loop, replaced by a classification-style loss on the pairs.
• The SFT stage, because preference training teaches format and tone without needing demonstrations of good outputs.

Choose one answer.

• Pairs of a ranked better and worse triage, which a reward model turns into a score for each output.
• A large pile of tickets with no labels, so the model learns the format from their statistical patterns.
• Reward scores for sampled outputs, which the model maximises with reinforcement learning against a KL penalty.
• Prompt and ideal-response pairs, where each ticket is paired with the exact triage output you want.

Choose one answer.

• 16,384 weights, about 0.39%, counting only one of the two matrices A and B at rank 8.
• 32,768 weights, about 0.78%, from A (8 x 2048) plus B (2048 x 8) against 4,194,304 weights in the layer.
• 65,536 weights, about 1.56%, reusing the 4096-wide count from the lesson's worked example without rescaling it.
• 33,554,432 weights, about 800%, from multiplying the full layer size by the rank of 8.

Choose one answer.

• It is scaled down, because the alpha-over-r factor shrinks the original weights before the update is added.
• It is slightly noisy, because A is initialised randomly and that randomness passes into the layer output at the start.
• It is identical, because B starts at zero, so the correction B x A contributes nothing until training changes B.
• It is zero, because only the adapter path runs and the frozen weights are bypassed during fine-tuning.

Choose one answer.

• Gradients and optimiser state are needed only for the small A and B matrices, not for the large frozen matrix.
• The frozen matrix is compressed to 4 bits during training, so it takes a quarter of the memory it used before.
• The forward pass skips the frozen matrix and runs only through the adapter, so each step does far less computation.
• Frozen weights cannot be overfitted, so no validation set is needed and the evaluation stage can be skipped.

Choose one answer.

• Distillation into the same 70B model, because training on its own outputs needs only the weights in memory.
• 16-bit LoRA, because the adapters are tiny and so the memory use is also tiny for the whole model.
• Full fine-tuning, because it updates every weight in place and so needs no additional memory for the model.
• QLoRA, which holds the frozen base at 4-bit (about 35 GB of weights) and trains small adapters on top.

Choose one answer.

• Slower training steps from dequantising weights, and a check on its own task, since quality parity is a reported result.
• Higher inference latency for every request, because the adapter must stay separate and is hard to merge afterwards.
• A noticeable quality loss of several points, because 4-bit storage discards information the task needs.
• A need to retrain the base model in 4-bit first, because adapters do not attach to a model that was trained at 16-bit.

Choose one answer.

• Full fine-tune twelve copies of the base model, because each customer's style needs every weight to be free to change.
• Prompt each request with all twelve style guides, since adapters need a separate copy of the base model for every customer.
• Train one small LoRA adapter per customer on the shared frozen base, and swap the right adapter in when serving.
• Train one rank-64 adapter on every customer's data together, because a higher rank lets one adapter hold all twelve styles.

Choose one answer.

• The eval set leaked into training, so discard the task result and build a new eval before doing anything else.
• The adapter was merged into the weights, so unmerge it to restore general ability and keep the task gains.
• The judge is biased toward the tuned model, so replace it and the general-ability results will recover.
• Forgetting from narrow training; lower the rank or learning rate, mix in general data, and re-run the suite.

Choose one answer.

• Remove the system prompt in production, since a tuned model does not need instructions to behave correctly.
• Retrain with examples formatted exactly as production formats them, including the system prompt and template.
• Raise the adapter rank, because a larger adapter can absorb differences between training and serving inputs.
• Increase the number of bare-question examples, because more of them will eventually cover the production template too.

Choose one answer.

• Overwriting the old adapter file on deploy, so there is a single current version to reason about.
• Versioned adapters stored with their data and hyperparameters, plus a rollback path to the previous adapter or base.
• Fine-tuning again from scratch on the combined data, because a fresh training run is the way to remove a regression like this one.
• Merging the adapter into the base weights at deploy, so the model is one file with no separate parts.

Choose one answer.

• Safety behaviour can weaken even with harmless data, so run safety checks on the tuned model, not only task metrics.
• Harmless data leaves safety behaviour untouched, so the base model's safety results carry over to the tuned model.
• Safety degrades when examples contain harmful content, so a review of the data content is enough to clear the release.
• Safety loss is specific to adapter methods, so switching to a full fine-tune removes the need for separate safety checks.

Choose one answer.

• The tune generalises well, since it has seen the kind of slip it will meet in service and handles them reliably.
• The score is still a fair estimate, since the model was trained with a different loss than the one used for scoring.
• It partly measures memorised examples, so it says little about live inputs and the eval must be rebuilt without the overlap.
• The score is inflated, but raising the pass threshold on the same set would correct for the memorised slips.

Choose one answer.

• Forgetting of general skills; add more examples of the task so the model recovers what it lost on the other tasks.
• Leakage of the held-out set into training; discard the held-out set and judge progress by the training loss alone.
• An under-trained model; raise the learning rate and run more epochs until the training and held-out curves meet again.
• The model is fitting its training examples at the cost of new inputs; halt at the best held-out score and broaden the data.