Retrieval-augmented generation quiz

Choose one answer.

• Fine-tune the model on the new policy text so its weights absorb the update and the old wording is overwritten.
• Nothing needs to happen, because the query step re-reads the original policy files each time a question arrives.
• Raise top-k so the new policy text is more likely to be retrieved alongside the older chunks that already exist.
• Re-ingest that document: re-chunk it, re-embed it and replace its old chunks, with no retraining of the model.

Choose all that apply.

• Embedding the user's question
• Splitting documents into chunks
• Building a prompt from the question and retrieved chunks
• Writing chunk vectors into the vector store

Choose all that apply.

• Each large chunk's single embedding now blends several topics, so it matches the question less sharply.
• Larger chunks put more irrelevant text into the prompt, which distracts the generator.
• The embedding model truncates text beyond 256 tokens, so the extra text in each larger chunk is no longer represented.
• The reranker is tuned for 256-token chunks, so its scores become random on anything larger.

Choose one answer.

• The vector index is too small to hold every row separately, so migrate to a larger index tier and re-run ingest.
• Tables should not be indexed at all, because numeric rows never embed meaningfully; remove them and rely on prose.
• Overlap between neighbouring rows is making them look alike, so set the overlap to zero and re-embed.
• The rows lost their column headers; keep the header with each row or rewrite each row as a full sentence.

Choose one answer.

• Add the document title and section context to each strip before embedding it, so the strip says which train it describes.
• Cut every strip into even smaller pieces so that each piece contains only one clearly separable fact to match.
• Tell the model at answer time to work out which train the strip must refer to, based on its general knowledge.
• Raise top-k to 50 so that low-ranked strips like this one are eventually included in the prompt for the model, with no extra work.

Choose one answer.

• The embedding model is biased toward restricted documents, so the team should re-embed everything with a larger model.
• Post-filtering removed the best matches; apply the permission filter inside the retrieval query itself.
• The system prompt should tell the model to ignore restricted documents, which makes the retrieval-side filter unnecessary.
• A shared answer cache across all users would reduce how many queries ever reach the permission filter at all.

Choose one answer.

• Chunk length in tokens, since shorter chunks are usually the more recent ones in a policy archive.
• The embedding model name, which tells the retriever which generation of the document each vector came from.
• The overlap percentage that was used when the document was first split into chunks at ingest time.
• Document date or version, saved at ingest, so the retriever can drop superseded chunks or prefer the current ones.

Choose one answer.

• Index at each size with other settings fixed, compare recall@k on labelled questions, and check prompt cost.
• Use 512, because it is the standard size most tutorials and framework defaults have settled on over the years.
• Pick the largest size, because more context per chunk always helps the generator find and quote the answer.
• Ask the LLM which chunk size it prefers, since it is the component that will have to read the retrieved chunks.

Choose one answer.

• Adopt it immediately, because any measured recall gain is worth any increase in per-query token cost.
• Discard the result, because recall@k cannot be compared across different chunk sizes in a fair way.
• Check end-to-end answer quality and per-query cost on the same questions before deciding whether the gain justifies it.
• Ignore the result, since an eight-point difference is within the noise of a retrieval experiment.

Choose one answer.

• Fine-tune the model on the policy documents every month so that the latest wording is stored in its weights.
• Put the full policy library into every prompt so that the model always sees the latest text and can cite it without retraining.
• Raise the model's temperature so it explores more of the policy wording it was trained on when it answers.
• RAG over the policy store, returning source ids, so a monthly update only means refreshing the changed documents.

Choose one answer.

• Making the model always reply in a strict house format, whatever the user asks it, across every conversation.
• Keeping answers current with a weekly price list, with citations, because weights go stale and cannot cite sources.
• Teaching a model a specialised classification task from many labelled examples drawn from the company's own history.
• Making a small model follow a company tone guide consistently, including phrasing, formality and sign-off conventions.

Choose one answer.

• Nothing visible, since vectors from any two embedding models are interchangeable once they have the same dimension.
• Only the new documents become unsearchable, because the old index cannot accept vectors produced by a newer model.
• Quality degrades because queries and old chunks no longer share a vector space; re-embed everything.
• The reranker automatically corrects the mismatch, because it reads the query and chunk text together at the end.

Choose one answer.

• Add BM25 and fuse the lists by rank, since raw BM25 and cosine scores sit on different scales and cannot be summed.
• Switch to a bigger embedding model, which will have enough capacity to memorise exact codes like this one.
• Lower the temperature of the generator so it sticks more closely to whatever the retriever returned.
• Add BM25 and sum its raw scores with the cosine scores, since a higher number is always the better match.

Choose one answer.

• Cross-encoders are less accurate than vector similarity, so they are only useful as a final tie-breaker.
• Rerankers can only accept a fixed input size of exactly 50 candidates, set by the way they were trained.
• Reranking replaces the vector index, so keeping the index after reranking is redundant work for the system and its users.
• Each query-chunk pair needs its own model call that cannot be precomputed, which is too slow over the whole corpus.

Choose one answer.

• The final prompt text alone, since it shows exactly what the model read when it answered.
• The answer plus a thumbs-down from the user, since that flags the failing request for review.
• Retrieved chunk ids with their scores, plus the final prompt, so you can see exactly what the model was shown.
• A nightly aggregate of recall@k over all requests, since it tracks retrieval quality over time.

Choose one answer.

• The generator's temperature setting, since a lower value would have stopped it quoting older content.
• Ingestion and index freshness: date or version metadata should have replaced or filtered out the superseded page.
• The reranker's model size, since a larger reranker would have scored the older page lower than the new one.
• The chunk overlap, since a larger overlap would have carried the cancellation notice into the neighbouring chunk.

Choose one answer.

• Generation, by tuning the prompt wording so the model asks itself whether the evidence is complete.
• Ingestion, because the passage must be missing from the corpus or parsed into unusable text by the pipeline during the last re-index.
• Evaluation, because the metric used to flag the failing query must be miscounting what was retrieved.
• Retrieval: chunking, the embedding, the hybrid setup, filters or query rewriting, since the passage never reached the prompt.

Choose one answer.

• Generation: the prompt, the model or noise in the context, because the evidence was present and still not followed.
• Retrieval, because the reranker should have placed the passage first so the model could not miss it in the prompt.
• Ingestion, because a document that is parsed correctly can still be indexed in an outdated state.
• Evaluation, because contradictions of this kind cannot happen when the golden set has been run.

Choose one answer.

• Send all twelve results plus the rest of the candidates into one very large prompt every time.
• Retrieve about 50 candidates and rerank them down to 5, so the rank-12 page can be sorted into the shortlist.
• Lower the generator temperature so the model is more likely to notice material that sits below the cut-off.
• Ask the model to read the lower-ranked results as well, since it can request chunks it knows are missing.

Choose one answer.

• A, because holding rank 1 in even one list always beats holding rank 2 in two lists.
• They tie, because RRF ignores any document that appears in only one of the lists being fused.
• B: 1/62 + 1/62 is about 0.0323, which beats A's single term 1/61, about 0.0164, so agreement across lists wins.
• It depends on the BM25 raw scores, which RRF rescales before it adds the reciprocal-rank terms together, per document.

Choose one answer.

• Cross-encoders are unable to judge identifiers such as part numbers or error codes in any query they see.
• The reranker model was too small, and a larger cross-encoder would have found the chunk outside the 50.
• Reranking increases recall as long as the shortlist is long enough to contain several chunks about the topic.
• Reranking only reorders its candidates, so add BM25 to bring the exact-token match into the shortlist at all.

Choose one answer.

• Rewrite it into a standalone question, which costs an extra LLM call and carries some risk of drifting from the intent.
• HyDE: embed an imagined answer instead of the query, which costs an extra LLM call to write that answer, on every query.
• Metadata filtering by date, which narrows the candidate set without needing any extra model call.
• A bigger chunk size, which keeps more of the earlier conversation attached to each stored passage.

Choose one answer.

• Add more paraphrases per query, so the extra searches cancel out the noise from the first ones.
• Remove retrieval from precise queries and let the model answer from what it has learned in training.
• Gate it: skip expansion for precise queries and use it only for short or ambiguous ones.
• Switch every request to HyDE, since writing a hypothetical answer cannot drift away from what the user meant.

Choose one answer.

• Search the shared index, then drop results whose tenant tag differs from the caller's before building the prompt.
• A tenant filter or namespace set by the server from the authenticated identity.
• Read the tenant from a field in the request body, so clients can choose which of their own tenants to search.
• Tell the model which tenant the user belongs to and instruct it to ignore chunks that belong to others.

Choose one answer.

• Cross-encoder reranking on its own, since it scores every candidate precisely against the query.
• A higher chunk overlap, so each chunk carries more of its neighbours' text and looks less alike.
• A smaller embedding model, which would be less likely to treat close paraphrases as separate matches, and so avoid duplicates.
• Maximal marginal relevance (MMR) or de-duplication, which trades relevance against redundancy among the chosen chunks.

Choose one answer.

• At the start or end of the context, because models tend to use the middle of a long prompt less reliably.
• In the middle, where it is surrounded and so protected by the supporting chunks on either side of it in the prompt.
• Anywhere, because position never matters for any current model once relevance has been established.
• Last only, because the model largely ignores whatever it reads at the beginning of a prompt.

Choose one answer.

• Build a hybrid-plus-rerank baseline, measure where it fails, and add a variant only for a failure you can name.
• Adopt agentic RAG with several tools, since it is the most capable design and so removes the need to diagnose.
• Add GraphRAG, Self-RAG and CRAG together so that every known class of failure is already covered up front.
• Fine-tune the generator on past answers, on the assumption that retrieval is already good enough for the corpus.

Choose one answer.

• Add HyDE, because writing a hypothetical answer first is the usual cure for retrieval that misses the target, in most pipelines.
• Introduce GraphRAG to model how the parts relate, so that part numbers can be found by traversal.
• Tune hybrid search so BM25 contributes exact-token matches to the fused ranking, since the failures are identifier misses.
• Let an agent loop with several retries until it happens to locate the part in some index.

Choose one answer.

• Agentic RAG with web, SQL and vector tools, so it can plan multi-step lookups for any question.
• GraphRAG over the whole document set, so that relations between topics are available to every answer in the whole corpus.
• Self-RAG, adopted by prompting the model to critique itself and to decide when retrieval is needed.
• Naive or advanced RAG, adding stages only if measurement shows a need, because the questions are simple and stable.

Choose one answer.

• Modular RAG has no retrieval step of its own and relies on the model's memory for the evidence.
• Modular RAG makes stages swappable modules joined by a router that can branch and loop.
• Modular RAG lets the LLM decide freely whether to retrieve at all, whereas advanced RAG never does.
• Advanced RAG requires a specially trained model, while modular RAG works with any general-purpose model.

Choose one answer.

• The hypothetical answer replaces the query in the search, so any error in it becomes a retrieval error.
• The embedding model cannot represent text that was generated by another model, so the vectors come out noisy.
• HyDE deletes or rewrites chunks in the index to match the guess, so the stored text itself drifts.
• HyDE disables keyword search entirely, so exact terms in the question can no longer influence the results.

Choose one answer.

• Nothing, because Self-RAG is a prompt pattern that works with the model you already deploy in production.
• Self-RAG is designed for image and chart retrieval, so plain text documents are out of scope for it.
• Self-RAG removes the vector index and so requires re-architecting the store before any prompt could work.
• It needs a specially trained model that emits critique signals; an LLM-prompted relevance grader is the no-training alternative.

Choose one answer.

• Parent-child retrieval, at the cost of larger prompts because whole parent sections are returned.
• Corrective RAG, which grades retrieval and falls back to a rewrite or web search, at the cost of a fallible grader.
• Adaptive RAG, at the cost of a router that can send queries down the wrong strategy for them.
• GraphRAG, at the cost of heavy indexing that has to be redone as the corpus changes over time.

Choose one answer.

• Embeddings are too inaccurate for tickets, so the team should adopt a bigger embedding model and retry the run.
• Top-k retrieval is too slow for that many tickets, so a cross-encoder reranker should be used instead.
• The answer depends on the whole corpus, so use corpus-level summaries such as GraphRAG, at the cost of heavy indexing.
• It struggles only because tickets are short, so merging tickets into longer chunks and raising top-k fixes it.

Choose one answer.

• Send all three through the same top-k vector search, which keeps the system simple and predictable.
• HyDE for (1), BM25 for (2), agentic RAG for (3), since each query then gets a different method.
• Use HyDE for all three, so each query is first turned into a plausible answer before it is searched.
• Exact-token match for (1), corpus-level summaries for (2), the cheap baseline for (3).

Choose all that apply.

• The model ignored the passage and answered from familiar training data.
• The model misread the passage, for example dropping a negation or picking the wrong figure.
• The query was embedded with a different model from the chunks, which breaks retrieval.
• The model added a plausible detail that the passage never stated.

Choose one answer.

• Agree, since retrieved context guarantees that every answer is grounded in the company's own documents.
• Disagree: RAG increases hallucination, because a longer prompt gives the model more room to invent details and drift from the question.
• RAG reduces hallucination, so measure faithfulness and abstention on a golden set and add checks such as citations.
• Agree for long questions, since extra detail in the query gives retrieval enough to ground every claim.

Choose one answer.

• The threshold should always be raised after a model change, because newer models produce tighter matches.
• Scores differ across embedding models, so the threshold must be recalibrated on labelled data for the new one.
• Retrieval gates cannot work with dense retrieval at all, so the gate should be removed in favour of reranking.
• The model forgot its grounding instruction after the embedding change, so it began refusing by default.

Choose one answer.

• Measure citation accuracy: check each claim against its cited chunk with a verification pass or a calibrated judge.
• Remove the citation requirement, since it clearly does not work and may give users false confidence.
• Ask for several citations per sentence, so a claim is more likely to be backed by at least one chunk in the prompt.
• Raise top-k so that the model has a larger pool of chunks from which to pick a supporting citation.

Choose one answer.

• A larger reranker, so that the best evidence is always sorted to the top of the prompt for the model.
• More overlap between chunks, so that every fact appears in at least two places in the index.
• A higher generation temperature, so the model explores alternatives and is less locked into one answer.
• An abstention path: a calibrated retrieval gate and an "I don't know" instruction, tested on unanswerable questions.

Choose one answer.

• Retrieval failure: the retriever picked a bad chunk, so the ranking stage needs tuning.
• Ingestion failure: the document was parsed wrongly, so the stored chunk text is not trustworthy.
• Generation failure: the model ignored or misread correct evidence that was already in the prompt.
• Source failure: the document is out of date, so the Platform 4 text no longer reflects the timetable.

Choose one answer.

• A pure generation failure, because the retriever returned results and so did its job.
• Retrieval missed, and no abstention rule stopped the model from answering fluently anyway.
• Normal behaviour, since a fluent answer shows the model already knows the material well.
• A reranking failure only, because the relevant chunk was ranked below the cut-off.

Choose one answer.

• A single score cannot name the failing stage; measure retrieval and generation separately, one change at a time.
• It is easy to act on: revert whichever of the two changes was made last, and the score will recover.
• Only human reviewers can detect regressions, so automated retrieval and generation metrics are not worth tracking.
• Always ignore drops smaller than 10 points, because anything below that is within the noise of any RAG metric.

Choose one answer.

• As proof the prompt is better, because the score went up and a golden set is meant to settle such questions.
• As proof the prompt is worse, because an improvement this small is more likely a sign of overfitting.
• As irrelevant, since golden sets should never be used for judging prompt changes in the first place.
• As inconclusive: enlarge the set, check which question flipped and why, then rerun before deciding on it.

Choose one answer.

• A single score cannot be computed automatically, so every answer would need to be graded by a person.
• Only generation matters, so a separate retrieval score would just repeat what the answer score already says.
• It cannot show whether retrieval or generation caused a miss, so you cannot tell which stage to fix.
• It is always lower than the retrieval score, so it makes a healthy system look worse than it is.

Choose one answer.

• Nothing changed overall, so the change is safe to ship and the stage metrics can be ignored.
• Faithfulness cannot rise unless recall also rises, so one of the two measurements must be faulty or mis-scored.
• Recall@5 should be dropped once accuracy is tracked, since accuracy already covers what retrieval does.
• Two effects are offsetting; investigate the retrieval drop before harder queries expose it as a failure.

Choose one answer.

• Recall@5 is about 0.67 and MRR = 0.375, since only the two found queries are averaged, (1/4 + 1/2) / 2, and the miss is skipped.
• Recall@5 is about 0.67 and MRR = 0.25, since two of three queries hit and (1/4 + 1/2 + 0) / 3 = 0.25, counting the miss as 0.
• Recall@5 = 1.00 and MRR = 0.25, since both found chunks were inside the top 5 and the miss does not count against recall.
• Recall@5 is about 0.67 and MRR = 0.33, since the found ranks 4 and 2 average to rank 3 and 1/3 is its reciprocal.

Choose one answer.

• About 0.58, because the sum 1/4 + 1 + 1/2 is divided by the three queries that found something.
• 0.75, because three of the four queries found a relevant chunk somewhere in the top results.
• 2.33, because the found ranks 4, 1 and 2 average to that value across the three queries that hit.
• About 0.44, because 1/4 + 1 + 1/2 + 0 = 1.75 and a miss still counts in the average over four queries.

Choose one answer.

• 0.50, the reciprocal of the rank 2 at which the first relevant result appears, so later hits are ignored.
• 1.00, because both relevant chunks were found somewhere inside the top 5 results, so nothing was missed.
• 0.40, because 2 of the 5 results in the top 5 are relevant, and precision@k is relevant results divided by k.
• 0.375, the mean of the reciprocal ranks 1/2 and 1/4 over the two relevant results.

Choose one answer.

• The source, the question or the gold label is wrong or ambiguous, so review them rather than the pipeline.
• Retrieval failure; tune chunking so the gold chunks come back higher in the ranking next time.
• Generation failure; change the prompt, because the model must have misused the evidence it was given.
• Lucky answer from memory: the model answered from its training data rather than from the retrieved text.

Choose one answer.

• It is not a concern: a correct answer is a correct answer, whatever route the model took to it.
• It proves the retriever is already strong enough, since the system reached the right answer without it.
• It came from the model's memory, not the corpus, so it may not hold for similar questions.
• It only shows that the golden set is too small, and a bigger set would remove the problem entirely.

Choose one answer.

• Questions an LLM wrote from the chunks, all answerable and never reviewed, so the set can grow quickly.
• Logged real queries labelled with gold chunks, covering several types, some unanswerable, plus a fresh set.
• Mostly the questions the team already knows the system answers well, so the report stays clean.
• One long list of very hard questions, since easy cases add little that hard ones would not reveal about the system.

Choose all that apply.

• Self-preference: judges tend to favour answers from their own model family.
• Verbosity bias: judges tend to reward longer answers.
• Over-fitting to the regression set.
• Instability: repeated judging runs give different scores, which is the pitfall shown here.

Choose one answer.

• Use a fresh set from production each time, since a frozen set goes stale as queries change over time.
• Use a frozen set, since fresh data adds noise and makes comparisons between runs harder.
• Both sets should hold identical questions, so any score gap comes from the system itself.
• Keep a frozen set for comparability and a rotating production set to guard against overfitting.

Choose one answer.

• Three changes at once, so the gain cannot be attributed, and one question of 40 is within noise.
• The set is too small for any experiment, so no comparison should ever be made on only 40 questions.
• Latency is a deployment concern and should only be examined after a quality gain is confirmed on a larger set.
• Changing several variables at once is faster and attributes just as well, since the metric is averaged over questions.

Choose one answer.

• Ship it, since a measurable quality gain usually outweighs a modest latency increase.
• Weigh the gain against the added latency and cost for this product, per slice, then decide.
• Reject it, since a latency increase in production is a regression whatever the quality gain.
• Judge it by the average end-to-end score alone, since stage metrics add nothing once that number exists.