Cheat sheet: GenAI system design

The method (nine links, in order): requirements, scale estimate, components, data flow, failure modes, evaluation, cost and latency, security, rollout. Estimate only the numbers that change a decision, show the arithmetic, and label every number illustrative.

RAG QA system: ingest (load, chunk, embed, store with allowed groups) plus query (resolve user, hybrid retrieve filtered by permissions, rerank, evidence gate, generate with citations). Filter permissions inside the retrieval query, re-check the final chunks against the source system, and scope caches by permission set. Abstain below a tuned evidence threshold. Worked example: 2,000,000 chunks × 1,024 × 4 bytes is about 8.2 GB; the LLM call, not retrieval, dominates cost.

Agent product: give each tool an autonomy level from its worst case (read, reversible write, approve first, never). Approval screens come from the structured call. Authorization lives in code. History is re-sent every step, so cost grows faster than steps: cap steps, cost and time. Executor-created idempotency keys, a same-key retry after a timeout (with a lookup by your own reference when keys may have expired), and compensation for multi-step failures. Grade final state and consistency across runs.

Content moderation: layers are a menu (input classifier, policy prompt, output check, human lane, logging and appeals). Choose serial, windowed or parallel output checks by harm severity, because streamed text cannot be un-shown. The threshold is a per-category business decision made with counts of both error types; when the false-positive rate is not far below the base rate, most blocks are false positives. Size the human lane from reviewer capacity, tune in shadow mode, keep rollback behind a flag.

Always volunteer in a GenAI design answer:

• Permission filtering inside retrieval
• An abstain path instead of a confident guess
• A cost and latency budget (tokens × calls)
• An evaluation plan that separates retrieval from generation, and an observability plan
• Guardrails against injected content, and approval for irreversible actions

30-second answers#

• Framework: requirements, estimates, components and data flow, deep dives (retrieval, evaluation, safety, cost), failures and rollout, trade-offs.
• Always say: how it is evaluated, how it fails, what it costs, and what changes at 10 times scale.
• Constraint changed: restate it, recompute the number it changes, name the component and the cost.
• Senior signal: failure modes and trade-offs raised without being asked.

Traps#

Jumping to components before requirements, no numbers, no evaluation plan, permissions added late or applied after generation, a single accuracy metric with no retrieval and generation split, inventing project metrics.