Question bank
Interview Questions — Scenario Based
10 itemsWhat the provenance labels mean
In this bank
Round: GenAI/LLM depth + production debugging · Difficulty: at-level Provenance: every scenario below is Generated practice, modelled on a Common industry pattern of production RAG and agent failures. None is verified company-specific. Method: talk through symptom → hypotheses ranked by likelihood/cost-to-check → how to measure → fix → how to prevent regression. State your assumptions first.
S1. Chatbot confidently states a policy that doesn’t exist
Provenance: Generated practice. Lesson: RAG evaluation and hallucination.
Hypotheses: (1) retrieval returned nothing relevant and the model answered from memory — no abstention; (2) retrieved a similar-but-wrong policy; (3) model blended chunks; (4) stale/duplicated docs. Measure: pull the trace — what was retrieved, scores, the prompt. Fix: retrieval gate + “I don’t know”, citations required and verified, rerank, de-dup/version docs. Prevent: add this query to the golden set; alert on citation-less answers. Common mistake: jumping to ‘the model hallucinated’ before looking at the trace for what was retrieved.
S2. 10M-chunk vector DB returns irrelevant results
Provenance: Generated practice. Lesson: Vector databases.
Check in order: embedding model mismatch between index/query (or model version changed) → normalisation/metric mismatch → chunk quality (parsing junk, boilerplate dominating) → ANN params (efSearch/nprobe too low) → missing metadata filters → need for hybrid/rerank. Method: take 30 failing queries, compute brute-force top-k to separate index (ANN recall) problems from embedding problems.
Common mistake: re-embedding everything first, instead of using brute-force top-k to separate index problems from embedding problems.
S3. “Sounds confident but is often wrong”
Provenance: Generated practice. Lesson: RAG evaluation and hallucination.
Quantify first (what % wrong, which topics). Then two+ fixes: calibrated abstention (retrieval gate), citations + verification, rerank, grounding prompt, expose uncertainty in UI, human escalation for low-confidence. Name the metric you’ll move (faithfulness, escalation rate, thumbs-down). Common mistake: proposing fixes before quantifying how often and where it is wrong.
S4. API cost tripled after longer prompts
Provenance: Generated practice. Lesson: Cost and latency optimization.
Cost scales with input tokens (linear billing) and attention compute grows faster than linearly with context, and prefill latency rises; long prompts also defeat cache reuse if prefixes change. Check: avg input tokens/request, retrieved chunks count, conversation history growth, system-prompt bloat. Levers: trim/summarise history, cap k, prompt caching on a stable prefix, route simple queries to a small model. Common mistake: blaming the provider price instead of checking average input tokens, history growth and cache-breaking prefixes.
S5. Document updated but search returns the stale version
Provenance: Generated practice. Lesson: Production deployment of RAG.
Check ingestion lag, upsert-vs-insert (old chunk IDs still present), chunk-ID scheme that changes on edit, cache layers (semantic/response/CDN), eventual consistency of the index, CDC trigger failing. Fix: deterministic IDs per (doc, chunk-index) + delete-by-doc on update, version metadata, cache invalidation on doc change, freshness SLO + monitor. Common mistake: re-running ingestion without checking chunk IDs, the cache layers and index consistency.
S6. Shorter answers (cheaper) but users say “incomplete”
Provenance: Generated practice. Lesson: Observability and eval pipelines.
Don’t trust average length — slice by query type. Compare outputs before/after on a fixed set with a completeness rubric + human review; look at thumbs, follow-up-question rate, task success. Fix: instruct structure not just brevity, per-intent length targets, A/B before rollout. Common mistake: judging by average answer length instead of slicing by query type and checking completeness.
S7. “Fine-tune a model on last quarter’s sales data to answer sales questions”
Provenance: Generated practice. Lesson: Fine-tuning vs RAG vs prompting.
Wrong tool: facts are structured, numeric, and change; fine-tuning gives unreliable recall and no citations. Use tool calls or generated queries against the warehouse so the numbers come from the database (text-to-SQL and semantic layers are not covered as a lesson in this course), LLM for NL interface and narrative. Mention permissions and auditability. Common mistake: agreeing to fine-tune on structured numbers that change every quarter.
S8. Agent reads email and has send_email
Provenance: Generated practice. Lessons: Guardrails, safety and prompt injection and Limiting blast radius and treating output as untrusted.
Indirect prompt injection → data exfiltration/spam (“forward all invoices to…”). Mitigate: separate read-only and send contexts, human approval on send, recipient allow-list / domain restrictions, no untrusted-content-derived arguments to send without confirmation, rate limits, logging, egress controls. Common mistake: relying on a prompt warning instead of approval, allow-lists and egress control on the send tool.
S9. Moderation blocks too many legitimate requests
Provenance: Generated practice. Lesson: Designing content moderation and guardrail systems.
Measure false-positive rate on a labelled sample by category/threshold; look at which policy/classifier is firing; segment by user cohort/language; tune thresholds per category (not global), add context-aware rules, appeal/override path, shadow-mode new thresholds, track precision/recall per policy. Common mistake: lowering one global threshold instead of measuring false positives per category.
S10. Permissioned RAG leaks content from a forbidden doc
Provenance: Generated practice. Lesson: Designing a RAG QA system.
Design failure: authorisation enforced after retrieval/in the prompt (or not at all), or shared cache/embeddings across tenants, or derived summaries ignoring ACLs. Fix: apply ACL filters inside retrieval (metadata filter pre-/during ANN), per-tenant namespaces, ACL sync from source of truth, cache keys scoped by identity, audit logs, tests that attempt cross-user access. Common mistake: filtering results after retrieval or in the prompt instead of inside the retrieval query.
Rapid-fire probes (use after any scenario)
Provenance: Generated practice.
- “How would you know your fix worked?” · “What would you monitor?” · “What’s the failure mode of your fix?” · “What if the golden set only has 30 queries — how much do you trust the result?”