Production: cost, serving and observability quiz

Choose one answer.

• Switch to the smallest available model, because model size is the main driver of both latency and cost.
• Record tokens in and out, time to first token and generation time per stage, then find the term that dominates.
• Turn on streaming everywhere, because users stop noticing a long wait once words start to appear on screen.
• Put a semantic cache in front of the model, because repeated questions are the cheapest requests to remove.

Choose one answer.

• Lower max_tokens from 300 to 150, so each response finishes sooner and the output bill falls a good deal.
• Add speculative decoding on the provider side, so that the output tokens are generated much faster.
• Stream the response, so that the cost of the 6,000 input tokens is spread across the whole answer.
• Retrieve fewer and shorter chunks and keep the stable prefix identical so prompt caching can reuse it.

Choose one answer.

• $0.017, made up of $0.012 for the input tokens plus $0.005 for the output tokens.
• $0.013, which prices all 6,500 tokens at the input rate of 2 dollars per million.
• $0.012, which counts the 6,000 input tokens and leaves out the generated tokens.
• $0.061, which swaps the two prices and applies the output rate to the input tokens.

Choose one answer.

• Halve the prompt length, since shorter prompts are the main way to speed up a response.
• Enable streaming, because the user will receive the whole answer sooner than before.
• Ask for a more concise format that cuts the answer to about 300 tokens.
• Cache the prompt prefix, because cached tokens remove most of the work for the request.

Choose one answer.

• Prompt caching only works for streaming requests, so every non-streaming call is processed from scratch.
• The prefix differs on every request, so there is never an exact match; the timestamp should move to the end.
• The cache is stored on the client, so the application must resend a cache id with each call.
• Cached prompts are only discounted for the output tokens, so input reads never show up in usage.

Choose one answer.

• The embedding model is too small, so upgrading it would stop similar questions from ever getting mixed up in the cache.
• The cache entries expired after five minutes, so that stale answers were served from an earlier user session on the site.
• The model generated a wrong answer, so the fix is a lower temperature at the time the answer is first created.
• Similar wording scored above the threshold, so a stored answer was reused for a question that differs in a key detail.

Choose one answer.

• Treat it as a routing gap for one query type, evaluate per slice, and perhaps send that slice to the large model.
• Accept the result, because an unchanged average accuracy proves the routed system is equivalent to the old one overall.
• Remove routing entirely, because any quality loss on one query type means model routing cannot be used safely.
• Add a response cache for legal questions, since a cached answer will replace the weak model output on repeat questions.

Choose one answer.

• Time to first token alone, because a shorter prompt is a latency change and the bill is already settled by token counts.
• Nothing more, because input tokens are billed below output tokens, so trimming them cannot change what an answer says.
• Answer quality on a labelled question set, because fewer chunks may have dropped the passage that held the answer.
• The cache hit rate, because a lower bill after a prompt change shows the stable prefix is now matching more often.

Choose one answer.

• No. Streaming should have reduced token usage, so a configuration error is the likely explanation for the bill.
• No. Streaming runs decode on a faster path, so total time should have dropped and something is misconfigured.
• Yes. The first token arrives earlier, but total generation time and the number of tokens billed are unchanged.
• Yes. Streaming moves some work to the client, so the server bill should have dropped by the cost of that work.

Choose one answer.

• A provider batch API makes each request finish faster, because requests in a batch share one pass through the model.
• A batch API trades speed for price: embedding 2 million documents overnight fits, an autocomplete box does not.
• Continuous batching on your own GPUs lowers latency for every request, whatever the traffic load happens to be.
• Batching applies only to self-hosted models, because hosted providers process each request on its own.

Choose one answer.

• Buy a GPU with twice the arithmetic throughput but the same memory bandwidth as now.
• Store the weights in 8-bit numbers, so each decode step reads about 7 GB instead of 14 GB.
• Shorten the prompt, because decode speed is set mainly by how many input tokens the user sent.
• Reserve a larger KV-cache pool, because decode is limited by how much cache memory is available.

Choose one answer.

• About 27, because the usable 40.8 GiB minus 13 GiB of weights leaves 27.8 GiB of cache.
• About 35, because the full 48 GiB minus the 13 GiB of weights leaves 35 GiB for the cache.
• About 40, because the usable 40.8 GiB divided by 1 GiB per request gives the count.

Choose one answer.

• Decode is bandwidth-limited, so quantizing the weights is the right first change to make for this symptom.
• The batch is too small, so a larger batch size would help every request finish faster than it does now.
• Requests are queueing or long prompts slow prefill, so check queue depth and add replicas or prefix caching.
• The KV-cache is too small for decoding, which would cause the text to appear in short bursts to the user.

Choose one answer.

• Server A, because it serves more requests per second and so produces more useful work.
• They are equal, because goodput and throughput measure the same thing for a fixed GPU.
• Neither can be ranked, because goodput needs token counts per request rather than request counts.
• Server B, because 8 x 0.95 = 7.6 good requests per second beats A's 10 x 0.6 = 6.

Choose one answer.

• Decode is slower at peak because weights are re-read from disk on every step, so each running request pays more per token.
• Answers are longer at peak, so every response takes more time to finish and its first token appears later.
• New requests wait for a slot, because a full KV-cache pool keeps arrivals queued before prefill starts.
• Quantization error builds up under load, so the server spends extra time per token checking its own output.

Choose one answer.

• Prefix caching, because the long prompt will be answered from stored results for the next user.
• Chunked prefill, which splits the long prompt so other users' decode steps run between chunks.
• Speculative decoding, since a draft model finishes the long prompt faster than the large one.
• A larger static batch, which gives the long prompt its own slot so it cannot disturb the others.

Choose one answer.

• Outputs follow the large model's own distribution, and the speed-up for most methods shrinks as server load rises.
• The draft model's wording leaks into answers, so the technique trades some answer quality for the speed gain.
• It gives the biggest gains on saturated servers, where draft tokens use the otherwise idle compute.
• It reduces KV-cache memory per request, which is why it lets a GPU hold more concurrent users.

Choose one answer.

• It returns a stored answer whenever a question repeats, so the model is not called again for that user.
• It makes every decode step faster, because the shared tokens no longer need their keys read again.
• It skips recomputing the shared prefix's keys and values, which cuts prefill work and time to first token.
• It lowers the memory the weights need, so a larger model can fit on the same GPU card without any other change.

Choose one answer.

• Ignore the tickets, because lower precision changes speed and memory but cannot change answers.
• Switch to a larger batch size, because the wrong numbers come from the scheduler rather than the weights.
• Turn on speculative decoding, because the draft model corrects arithmetic mistakes in the main model.
• Compare both versions on an eval set with numeric cases, then decide per task whether the loss is acceptable.

Choose one answer.

• Prefix caching, because shared leading tokens would then reuse the reserved memory of earlier requests.
• Quantizing the weights, because smaller weights shrink the reservation made for every request.
• Chunked prefill, because splitting long prompts into pieces lets the server reserve less for each of them.
• PagedAttention, which allocates KV memory in small blocks as tokens arrive, so short requests waste little.

Choose one answer.

• About $0.56, which is the full-load price of 2.40 / 4.32 per million tokens served.
• About $2.22, because the GPU makes 1.08 million tokens an hour and 2.40 / 1.08 is that price.
• About $0.14, because a quarter of the full-load price is what low utilization costs.
• About $9.60, because the 2.40 hourly price divided by the 25 percent utilization gives the answer.

Choose one answer.

• Use a hosted API, since cost follows usage, idle time is free and the team runs no serving stack.
• Use a hosted API and add a reserved GPU as a standby, so that spikes do not wait for capacity.
• Self-host a quantized open model, since quantization narrows the capability gap to frontier models.
• Self-host with speculative decoding, since it removes the need to scale for traffic peaks.

Choose one answer.

• Total latency and cost for the request, because a slow or expensive request is usually the one that went wrong.
• The thumbs-down flag and the user id, because they identify who was unhappy with the answer.
• The retrieved chunk ids with scores and the final prompt, to see if the right passage was supplied.
• The model name and temperature, because those two settings determine whether a policy gets quoted correctly.

Choose one answer.

• Version fields for the embedding model, chunker, index and prompt, so quality can be split by version.
• The raw response text, because wording differences between weeks would point straight at the cause.
• The output token count, because a change in retrieval always shows up as a change in length.
• The request timestamp, because dates alone are enough to locate the faulty component.

Choose one answer.

• Retrieval scores slowly drifting over several weeks, since only a labelled test set can see a distribution change.
• A rising thumbs-down rate over several weeks, since slow trends cannot be seen on a live dashboard.
• A cost-per-request increase after prompts grew, since cost can only be computed in an offline report.
• A prompt edit that breaks a rare intent used by only a few users, so that no aggregate metric visibly moves.

Choose one answer.

• Thumbs-down data was not recorded, since flat rates mean the signal was dropped by the logger.
• Few users rate answers and those who do are unrepresentative, so the signal is weak and slow.
• The rate is measured per session, so a few bad answers are averaged away across many good ones.
• Feedback lags by a few days, so the vote count will rise later and the quality review was premature.

Choose one answer.

• Do not ship yet. One point is within noise of about 2 points, and the refund drop needs its failing cases read.
• Ship it, because the overall score went up and a slice of twenty cases is too small to matter.
• Ship it and watch the refund slice, because the average is the metric that predicts overall user satisfaction.
• Reject it because any slice drop means a prompt change fails, whatever the other slices do in the set.

Choose one answer.

• Fix the prompt for that exact question and move on, since the golden set should stay a fixed benchmark.
• Delete the trace to protect the customer, since stored mistakes are a legal and reputational liability.
• Label it, redact personal data, and add it to the golden set with a slice tag before fixing.
• Wait for several more thumbs-down on the same topic before adding it, to avoid overfitting.

Choose one answer.

• Good news, since a model that says "I don't know" more often is hallucinating less in its answers.
• Behaviour changed, perhaps retrieval got worse or the prompt became more cautious, so compare traces.
• Nothing to act on, since abstention is not an error and the error rate is stable, so alerts are fine.
• A latency problem, since abstentions are returned when the model times out waiting for retrieval.

Choose one answer.

• Traces are too short to be useful test cases, since they only contain the final answer text.
• Golden sets must be written by hand, since production data cannot serve as valid test cases for a model.
• The traces would be too noisy to label, because most requests are duplicates of one another.
• They may hold personal data and secrets, so redaction, access control and retention apply first.

Choose one answer.

• A stricter latency alert, because a wrong answer takes longer to produce and lifts p95 above its usual baseline.
• Alerts on a sampled groundedness check and on abstention and empty-retrieval rates, against the system's own baseline.
• A lower sampling temperature, which would have kept the wrong answers from varying between one request and the next one.
• A larger golden set run before each release, so the offline gate would have caught the failure before users did.

Choose one answer.

• None: run the check in parallel with streaming and it will block a bad answer before the user sees it.
• Streaming raises token cost, so the check must run on a cheaper model to keep the economics workable.
• Streaming is incompatible with any output check, so the product must choose between streaming and guardrails.
• Streamed text cannot be checked first, so choose check-first, stream-first or chunked checks.

Choose one answer.

• Run ingestion in the same pool but at a lower priority so that serving keeps working at full speed.
• Run ingestion on a separate pool, autoscaled on queue depth, so serving is left untouched.
• Skip nightly re-embedding and rebuild the index only when users complain about stale answers.
• Add more vector database replicas, since the spikes must come from slow vector search.

Choose one answer.

• Deletes and permission changes are not propagated to the index, so stale chunks stay retrievable.
• The embedding model has memorized the document, so only retraining the model can remove the content.
• The LLM cached the document in its weights during earlier answers, so it must be reset.
• The reranker is promoting the document, so its scoring needs to be retuned to drop it.

Choose one answer.

• Keep the old vectors and apply a small score correction, since both models embed exactly the same text.
• Re-embed documents gradually in place, so that some old and new vectors mix during the transition period.
• Build a new index with the new model, pass the eval gate, then switch the alias and query model together.
• Re-embed only the documents that users search most, since the others will not be queried.

Choose one answer.

• A larger embedding model, so the revised wording of each page is matched more closely to the questions users ask.
• A reranker that prefers newer pages, so withdrawn and superseded documents sink below the current ones in every ranking.
• Change detection with incremental updates, deletes that propagate and an alert when ingestion lag passes its target.
• A shorter response-cache lifetime, so that stored answers expire sooner and are regenerated from the revised pages.

Choose one answer.

• About 10 GB, from 10,000,000 x 1,024 x 1 byte per value.
• About 41 GB, from 10,000,000 x 1,024 x 4 bytes per value.
• About 20 GB, from 10,000,000 x 1,024 x 2 bytes per value.

Choose one answer.

• About 30 percent: cost falls from $0.0067 to $0.0047 because output and other input stay.
• About 50 percent, because half of the chunks were removed from the request entirely.
• About 43 percent, from the input-only drop of 0.0020 out of an input cost of 0.0047.

Choose one answer.

• Yes, because 2.9 s is under 3 s, so the sum of stage values proves the end-to-end p95.
• It depends on the slowest stage only, so the 1.5 s model call is the real p95 of the request.
• No, because streaming adds the whole generation time to the 2.9 s and pushes it past 3 s.
• Unknown until you measure the end-to-end p95 directly, since percentiles of stages do not add up.

Choose one answer.

• Cache misses will rise, because two different tenants rarely ask the same question in the same words.
• Answers go stale after a document is updated, since an embedding key cannot expire on a content change.
• Embeddings leak the original question text, so anyone with cache access can read it back directly.
• Another tenant's similar question can return an answer built from documents they cannot see.

Choose one answer.

• Skip retrieval and let the model answer from memory, since retrieval is the slowest step.
• Raise the reranker timeout so every request waits for it, since reranking is required for correct answers.
• Skip rerank after a timeout and serve hybrid results, which is a small and known quality loss.
• Switch to a larger LLM, since a stronger model can compensate for context that is not reranked.

Choose one answer.

• The model sees restricted text and a prompt instruction cannot reliably keep it from reaching the answer.
• The instruction is too long, so a shorter prompt would make the model follow the group rule more consistently.
• The prompt is fine, but the reranker should down-rank restricted documents so the model is less likely to use them.
• The flaw matters for multi-tenant products, because a single-company assistant has no users who differ in access.

Choose one answer.

• Roll back through the alias or flag, then use traces to find which queries lost their context.
• Continue to full rollout, because passing the offline gate proves the change is safe.
• Raise the canary share to 50 percent to collect more data before deciding on the change.
• Keep the canary as is and wait for the thumbs-down rate, since abstention is not a quality metric.

Choose one answer.

• A prompt is configuration, not code, so it needs no review and the only issue is recording who edited it.
• It skips versioning, the eval gate and staged rollout, so a regression is hard to catch or undo.
• Prompts are cached by the provider, so an edit is not applied until the cache entry expires.
• Prompts are too fragile to change in production, so behaviour changes should be made by fine-tuning instead.

Choose one answer.

• It is safe, because a reranker only reorders candidates and so it leaves the answers users receive unchanged.
• It needs only a latency check, since quality was already measured when the reranker vendor published its results.
• It goes through the same gate as any change: offline eval, shadow, canary, with the old version ready for rollback.
• It should be released to all users at once, so that the old and new rankings are never mixed in one conversation.