GenAI system design quiz

Choose one answer.

• About 2.3 GB, from 3,000,000 chunks times 768 dimensions at one byte per value.
• About 9.2 GB, from 3,000,000 chunks times 768 dimensions times 4 bytes per value.
• About 920 MB, from 300,000 documents times 768 dimensions times 4 bytes per value.
• About 18.4 GB, from 3,000,000 chunks times 768 dimensions times 8 bytes per value.

Choose one answer.

• Shard the index across several nodes now, so the system can grow without a later migration.
• Use one node with a replica for availability, and revisit sharding when the corpus grows a lot.
• Buy GPU capacity for the vector search itself, because retrieval latency dominates at this size.
• Precompute answers for every likely question, since the query volume is too small to justify live generation.

Choose one answer.

• About $0.0087, because all 2,900 tokens are billed at the input price of $3 per million.
• About $0.0078, because only the 2,600 input tokens count and generation is not billed separately.
• About $0.0399, because input is billed at $15 per million and output at $3 per million.
• About $0.0123, because input costs $0.0078 and output costs $0.0045 in a single query run.

Choose one answer.

• Add a keyword index and fuse it with the vector results, so exact tokens can still be matched.
• Switch to an embedding model with more dimensions, so each code gets a more detailed vector.
• Make chunks much larger, so the code always appears next to a long explanation of its meaning.
• Raise top-k to 50 and pass all candidates to the model, so the right article is likely included.

Choose one answer.

• In the system prompt, by telling the model to say it does not know, with no other check.
• After generation, by refusing any answer that lacks a citation to a retrieved chunk.
• On the reranked evidence, with a score threshold tuned on questions that have no answer.
• From user feedback, by hiding answers that receive repeated thumbs-down ratings later.

Choose one answer.

• A second model pass that re-reads the answer and corrects its own citations without consulting the retrieved set.
• A larger top-k, so more ids sit in the prompt and an invented id is more likely to match a real chunk.
• Removing citations from answers, because an id produced by the model cannot be trusted and plain answers are simpler to evaluate.
• A comparison of every cited id with the chunks placed in the prompt, flagging or dropping answers that cite outside it.

Choose one answer.

• Both return the same five passages, because the filter removes the same documents either way.
• The post-filter leaks the three documents, because the database has already shown them to the user.
• The in-query filter returns five readable passages, while the post-filter ends with only two left.
• The post-filter is better, because the search runs once over the full index with no extra constraint.

Choose one answer.

• Re-check the final chunks against the source system's permissions, then build the prompt.
• Add a line to the system prompt reminding the model to respect any recent group changes.
• Clear the answer cache once a night, so that old answers expire before the next working day.
• Trust the index metadata, because it was copied from the source system at ingest time anyway.

Choose one answer.

• Embeddings cannot be reversed, so stored answers are safe to share between users of the product.
• The cache hit rate drops, because similar questions from different users rarely land close together.
• A user can receive an answer built from documents that another user alone was allowed to read.
• Answers become stale, which is the main risk, and an expiry time on entries resolves it.

Choose one answer.

• The embedding model, since falling quality usually means the vectors need to be rebuilt.
• The chunk size, because smaller chunks would put the gold passage higher in the ranking.
• Top-k, because retrieving more passages would raise the share of claims backed by evidence.
• Generation: the new prompt, the context layout, or how the model handles that evidence in use.

Choose one answer.

• An ingestion-lag alarm that compares the newest indexed change with the freshness target.
• The golden-set recall dashboard, because stale data would lower retrieval quality on the golden set.
• The p95 latency dashboard, because a stalled pipeline would make queries slower for everyone.
• User complaints, because people notice quickly when an answer is out of date.

Choose one answer.

• Run it automatically, since the user already asked for the task and the call is logged for later review.
• Run it only after the user approves the exact recipient and body, because there is no undo.
• Run it automatically whenever the model reports high confidence in the draft it produced for this email.
• Remove the tool entirely, because irreversible actions do not belong in an agent product.

Choose one answer.

• The model's plain-language description of what it plans to do, written for the user to read.
• The full conversation so far, so the user can judge the request in its context.
• The structured call itself, such as payee, amount and dates, rendered by your own code.
• A risk score produced by the model, with an approve button shown when the score is low.

Choose one answer.

• 10,000 tokens, from 5 steps each sending the 2,000-token base prompt only.
• 22,000 tokens, from the base prompt plus 800 tokens times 1+2+3+4+5 additions.
• 26,000 tokens, from 5 steps each sending the final step's 5,200 tokens.
• 18,000 tokens, from inputs of 2,000, 2,800, 3,600, 4,400 and 5,200 tokens.

Choose one answer.

• It about doubles, from 7,000 to 14,000 tokens, since the step cap doubled from 4 to 8.
• It grows about 3.1 times, from 7,000 to 22,000 tokens, as history is re-sent.
• It grows exactly four times, from 7,000 to 28,000 tokens, as steps squared.
• It rises by about 2,000 tokens, because only the four new steps add content.

Choose one answer.

• Retry with the same key, which returns the first result if charged, or use your own reference.
• Retry at once with a fresh key, so the provider processes it as a clean new attempt, quickly.
• Ask the model to decide whether to retry, since it saw the error message and the context.
• Report failure to the user and stop, since an unknown outcome should not trigger more calls.

Choose one answer.

• Wrap both calls in one database transaction and roll the transaction back on failure.
• Keep the hotel and let the agent retry the flight payment until it finally succeeds somewhere.
• Run the hotel step's compensating action, such as a cancellation, and tell the user why.
• Leave the hotel as it is and explain to the user how to cancel it by themselves later.

Choose one answer.

• The key was too short, so two different requests collided and the provider treated them as separate tickets.
• The key came from the model, so each retry looked new; the executor should make one key per approved intent and reuse it.
• The key should be dropped, because a timeout is not an explicit error and a plain retry after a timeout is safe enough on its own.
• The retry came too soon, because a duplicate usually means the first request was still running when the second began.

Choose one answer.

• A system-prompt line telling the model to ignore instructions found inside web pages.
• A keyword filter that rejects tool results containing the word 'email' or 'passport'.
• A second model call that asks the first model whether its own plan seems suspicious.
• Approval before sends: the screen shows the unrequested recipient, so the user can decline it.

Choose one answer.

• Query the payment records for each run after it ends, and count charges for that task.
• Read the final message of each run, and count the runs that say the booking succeeded.
• Count steps per run, and flag runs above the step cap as likely double charges.
• Ask a judge model to rate whether the transcript sounds like a careful, correct booking.

Choose one answer.

• Stop the loop, return what it has with the reason, and escalate or ask the user.
• Raise the step cap for this task so the model has more room to find its way.
• Retry with exponential backoff, since repeated calls usually mean a transient tool error.
• Continue, but raise the sampling temperature so the model tries a different action.

Choose one answer.

• Stream immediately and run the classifier alongside, then retract the text if it is flagged.
• Check every 20 tokens as they stream and release each checked window with a short lag.
• Generate the whole answer, classify it, and release it to the user only once it passes.
• Skip the output check and rely on the input classifier, since harmful requests were already blocked.

Choose one answer.

• Displayed text may already have been read, copied or captured before the retraction happens.
• Streaming cannot be cancelled once begun, so a check running alongside the stream cannot work.
• Running the classifier alongside generation doubles the number of model calls and so the cost.
• Parallel checks are slower than serial ones, so the user waits longer for a safe answer.

Choose one answer.

• 95%, because recall of 95% means that share of all blocked messages is a real violation.
• 0.5%, because the false-positive rate describes how often a block turns out to be a real violation.
• 72%, because 9,980 of the 13,780 blocked messages were benign and wrongly blocked.
• About 28%, because 3,800 caught violations out of 13,780 total blocks is the precision.

Choose one answer.

• The ML team, by picking whichever threshold maximises the F1 score on the labelled set.
• The product owner and policy team, weighing missed harm against refused honest users.
• The classifier vendor, by keeping the default threshold in its documentation for all customers.
• The review team lead, by choosing the threshold that keeps reviewers' queues full all day.

Choose one answer.

• One global threshold at the strictest level, so every category gets the maximum protection.
• One global threshold set from the average score of traffic, so the block rate stays steady overall.
• A strict threshold and human lane for the severe category, lenient thresholds for the rest.
• Thresholds taken from the classifier's own calibration, so little policy input is needed.

Choose one answer.

• Precision 0.9 and recall 0.75, found by dividing 90 by the 100 violations and 90 by the 120 blocked.
• Both 0.75, found by dividing the 90 real violations by the 120 blocked messages for each measure.
• Precision 0.75 and recall 0.9, found by dividing 90 by the 120 blocked and 90 by the 100 real violations.
• Both 0.9, found by dividing the 90 real violations by the 100 real violations for each measure.

Choose one answer.

• 36 reviewers, from 3,600 items divided by 100 items per hour.
• 6 reviewers, since each handles 700 items a day and 3,600 / 700 is about 5.1.
• 5 reviewers, since 3,600 / 700 is about 5.1 and fractions can be rounded down.
• 4 reviewers, since 3,600 / 900 is 4 if each works a nine-hour shift.

Choose one answer.

• A clear refusal without internals, an appeal that reaches a person, and a stored label.
• A message naming the exact rule and score that fired, so the user can rephrase it.
• Allow the message on the second attempt, to limit frustration from false positives.
• Drop the message silently and log it, so that users do not argue with the system.

Choose one answer.

• Reviewers cannot judge messages the classifier allowed, since only blocked ones are reviewable.
• A random sample over-represents benign messages, and no weighting could correct for that.
• Recall equals precision at the chosen threshold, so a separate estimate adds no new information.
• Misses are about 0.01% of passed messages, so a 5,000-message sample holds about 0.5 of them, on average.

Choose one answer.

• Wait a week for more data, since a short period is too small to show whether the change helped.
• Roll back through its flag, then break the block rate down by category and by traffic source.
• Raise the threshold further, so that the block rate falls back toward its earlier level.
• Retrain the classifier at once on the new blocked messages, so it learns what to allow.

Choose one answer.

• Switch to a larger model, since confident wrong answers point to a weaker generator.
• Check whether the gold passage was retrieved and put in the final prompt, then judge generation.
• Lower the sampling temperature, since randomness is the usual reason for confident wrong answers.
• Rewrite the system prompt to demand more caution, since the model should say it is unsure.

Choose one answer.

• The total drops to 2.9 seconds, because tokens are produced while the user is reading.
• The first token still arrives at 5.9 seconds, since streaming changes only the answer's formatting.
• The total rises to 8.8 seconds, because streaming adds a delivery step after generation ends.
• The first token appears at 2.9 seconds, while the full answer still completes at 5.9 seconds.

Choose one answer.

• About 100%, since the number of retrieved chunks doubled and the cost follows it.
• About 77%, since input tokens rose from 2,600 to 4,600 and cost follows input tokens.
• About 49%, since the cost goes from $0.0123 to $0.0183 once output is counted too.
• No change, since chunks come from your own database and are not billed as model tokens.

Choose one answer.

• Fine-tuning is mainly a cost question, and a retrained model is cheaper per answer than retrieving pages on each request.
• A fine-tuned model would absorb each change as it happens, so the timetable stays current without any further steps.
• RAG removes hallucination on timetable questions, so the answers need no citations and no separate evaluation plan.
• Retrieval reads the current page at query time, so a changed timetable needs a re-index, not a retraining run.

Choose one answer.

• Say you are running over, close the data flow quickly, and state the order of what remains.
• Keep refining until the diagram is complete, since a half-finished design is the larger risk.
• Skip evaluation, because a strong architecture implies the system will work as intended.
• Ask the interviewer for ten more minutes, so that nothing has to be shortened.

Choose one answer.

• Recompute the index size first, since more traffic means more documents to store.
• Recompute the peak query rate, then its effect on replicas, rate limits and LLM cost.
• Say the design already scales, since the components chosen are standard in the industry.
• Start again from requirements with the new traffic figure and redraw the full architecture.

Choose one answer.

• Weak, because naming tools without any explanation shows no real understanding of them.
• Solid, because the pipeline is correct and the order of the stages shows working knowledge.
• Developing, because the boxes are correct but no single choice has any reason attached to it.
• Strong, because a correct pipeline is the hardest part and the rest can be taught later.

Choose one answer.

• A scale-estimate drill with worked arithmetic, repeated before the next full mock run starts.
• A rapid-fire round on architecture, since that is the dimension where Solid can become Strong.
• Another full mock at once, since only a complete run reveals whether the weakness is real.
• A debugging drill on failure modes, since Developing is the next level up from the lowest.

Choose one answer.

• Give a plausible figure from similar systems, since interviewers expect a number from you.
• Quote a vendor's published latency benchmark as the result your system achieved.
• Change the subject to accuracy, which you did measure, and return to latency only if asked again.
• Say you did not measure it, give the design target as a target, and say how you would measure.

Choose one answer.

• Name a real failure, how you found its cause, and what you changed or would change next.
• Say nothing major failed, since the project met its goals and the details are covered elsewhere.
• Describe a minor cosmetic issue, so that the answer sounds honest while risking little exposure.
• Blame an external vendor outage, since it was outside your control and cannot be second-guessed.

Choose one answer.

• Give the remembered figure as the new latency, since a rough real number is better than saying nothing.
• Say you ran it once at a quiet time with no record, so it is unmeasured under load, and add how you would measure.
• Quote the latency target from the design as the current latency, since that is the number the team promised.
• Decline to comment until a full load test has been run, since a partial answer might be quoted against you.

Choose one answer.

• Z, X, Y, W, because Z is first in the second list and a top rank outweighs X's ranks of 1 and 2.
• X, Z, Y, W, because X scores 1/61 + 1/62 = 0.0325 and Z scores 1/63 + 1/61 = 0.0323 in total.
• X, Y, Z, W, because the first list sets the order and the second list is only used to break ties.
• X, Z, W, Y, because a document from the second list is ranked above a single-list match from the first.

Choose one answer.

• P is first in one list only, and Q is second in both lists, so P ranks above Q in the fused list.
• P is first in one list only, and Q is fifth in both lists, so P ranks above Q in the fused list.
• P is first in one list only, and Q is 70th in both lists, so P ranks above Q in the fused list.
• P is first in one list only, and Q is tenth in both lists, so P ranks above Q in the fused list.

Choose one answer.

• The design prompt they rehearsed last week, so that the new score can be compared with the old one.
• A timed coding set from Drill D, since writing code under time pressure shows how well they organise their thinking.
• A partner who hints whenever the learner stalls, so that the session still finishes all of the blocks.
• A Drill C prompt they have not seen, with a partner who only asks questions and changes one constraint.