Module deck
GenAI system design flashcards
Reviewing needs JavaScript. You can read every card in the list below.
Loading your cards...
Back to the module32 cards in this deck
Read all 32 cards as a list
Designing a RAG QA system
- Front
- In a permission-aware RAG system, where does the permission filter belong?
- Back
- Inside the retrieval query, using allowed groups stored as chunk metadata, so restricted text never reaches the prompt. Never ask the model to withhold something it was already shown.
- Front
- What should a RAG system do when the retrieved evidence is weak?
- Back
- Compare the best reranked score with a threshold tuned on unanswerable questions, and say it found nothing instead of generating a confident guess.
- Front
- Why might hybrid (keyword plus vector) search beat vector-only search?
- Back
- Vector search matches meaning but can blur exact strings such as error codes and product names; keyword search matches those exactly. The cost is a second index to keep in sync.
- Front
- Name degraded modes for a RAG system.
- Back
- Retriever down: keyword-only search. Reranker down: serve hybrid results. LLM down: fallback model or cited snippets only. Stale index: freshness warning, or refuse for sensitive sources.
- Front
- How do you estimate raw vector storage?
- Back
- Chunks × dimensions × bytes per value. Example: 2,000,000 × 1,024 × 4 bytes is about 8.2 GB before index overhead (illustrative numbers).
- Front
- In a typical RAG query, what dominates cost and what does a retriever cost?
- Back
- The LLM call dominates: retrieved chunks make up most of the input tokens. Retrieval and embedding are small by comparison, so cutting top-k is a first lever (and a quality trade).
- Front
- How do you close permission sync lag and cache leaks?
- Back
- Re-check the final few chunks against the source system before building the prompt, and key any answer cache on the user's permission set.
Designing an agent product
- Front
- Why does agent cost grow faster than the number of steps?
- Back
- Each step is a model call that re-reads the growing history. With a 2,000-token base and 800 tokens added per step, 6 steps send 24,000 input tokens, not 12,000 (illustrative).
- Front
- Which controls make an irreversible agent action safe?
- Back
- Approval of the exact action, spend and rate limits, idempotency keys, audit logs, compensation where an undo exists, and least-privilege credentials.
- Front
- How do you assign autonomy to an agent's tools?
- Back
- By worst case: read runs freely, reversible writes run and can be undone, irreversible or external actions need approval, and the most dangerous tools do not exist.
- Front
- A payment call times out after possibly charging. What does the executor do?
- Back
- The executor retries with the same idempotency key it created, so a charge that happened is returned, not repeated. If keys may have expired or the provider lacks support, look up by your own reference first. A new key means a new charge.
- Front
- What is compensation in a multi-step agent task?
- Back
- A cancelling action for each side-effecting step (cancel the hotel), run in reverse order when a later step fails. Actions with no undo need approval first.
- Front
- How do you grade an agent that says it finished?
- Back
- Check the final state in the system of record (was the user charged once?), plus trajectory and consistency across repeated runs, not the agent's own claim.
Designing content moderation and guardrail systems
- Front
- What layers can a moderation system combine?
- Back
- An input classifier, a policy prompt, an output check, a human review lane, and logging with appeals. They are a menu chosen by risk, not one fixed order.
- Front
- Is the moderation precision and recall threshold a technical or business decision?
- Back
- Business: engineering supplies the counts of both error types, and the product owner and policy team choose the point, per category, from the product's risk.
- Front
- Serial, windowed or parallel output moderation?
- Back
- Serial (buffer then release) for severe harm, windowed streaming for mixed risk, parallel with retraction for low severity. Streamed text cannot be un-shown.
- Front
- Why are most blocks false positives when violations are rare?
- Back
- When the false-positive rate is not far below the base rate, the false positives outnumber the true catches. Example: 4,500 caught versus 9,950 honest blocked is 4,500 / 14,450, about 31% precision (illustrative).
- Front
- What is shadow mode in moderation tuning?
- Back
- Run a new threshold or classifier on live traffic, record what it would do, act on nothing, and compare with labels. Sample near the threshold, because random samples of passed traffic hold almost no misses.
- Front
- How do you size the human review lane?
- Back
- Items in the borderline band per day divided by one reviewer's daily capacity, rounded up. 2,000 items at 960 per reviewer per day is 2.08, so 3 reviewers.
RAG interview drills and mock
- Front
- Five things to volunteer in a GenAI system design answer?
- Back
- Permission-filtered retrieval, an abstain path, a cost and latency budget, an evaluation and observability plan, and guardrails against injected content.
- Front
- What do you do in the first five minutes of a GenAI design answer?
- Back
- Clarify users, scale, latency target, accuracy bar, cost ceiling and permissions; state assumptions and name what is out of scope.
- Front
- Quick pattern for a token-cost estimate?
- Back
- (Input tokens × input price + output tokens × output price) × requests per day; then name the lever that cuts it most and its quality cost.
- Front
- How do you evaluate a design before launch?
- Back
- A golden set with unanswerable cases, retrieval and generation metrics scored separately, a red-team for injection and leaks, a load test of p95, and shadow traffic.
- Front
- What breaks first at 10 times scale?
- Back
- It depends on what grows. More chunks hit index memory; more queries hit LLM rate limits and cost; more updates hit ingestion throughput. Recompute the changed number first.
- Front
- What is the senior-level differentiator in a design round?
- Back
- Raising failure modes, limits and trade-offs without being asked, and tying choices to numbers or metrics.
- Front
- How do you respond when the interviewer changes a constraint?
- Back
- Restate it, name the earlier number it changes, recompute that number, then say which component changes and what the change costs.
Coding, project defense and mock rounds
- Front
- What do you do when asked for a number you never measured?
- Back
- Be honest that it is unmeasured, separate a target from a result, and name the measurement you would add, such as a per-stage trace.
- Front
- When does a rank-1 document from one list beat a document with the same rank in both?
- Back
- With k = 60, when that shared rank r is above 62: 1/61 beats 2/(60 + r). At r = 62 they tie. Fusion rewards agreement but does not guarantee it wins.
- Front
- Can a document in only one list beat a document in both lists under RRF?
- Back
- Yes. With k = 60, first in one list scores 1/61 = 0.0164, while 70th in both scores 2/130 = 0.0154. Agreement helps, but it does not override a very weak rank.
- Front
- How do you label a number in a project defense?
- Back
- Measured, estimated or unknown. For an unknown, say so and name the measurement you would add.
- Front
- Four behavioural stories to prepare for a GenAI role?
- Back
- A quality regression you diagnosed, a latency versus quality trade-off, a time you pushed back on an LLM where something simpler worked, and a time you managed expectations about hallucinations.
- Front
- What are the three mock rounds?
- Back
- Round 1: questions (Drills A and B). Round 2: an unseen design prompt (Drill C). Round 3: project defense plus coding (Drills D and E).