Module deck

Production: cost, serving and observability flashcards

Reviewing needs JavaScript. You can read every card in the list below.

Loading your cards...

Back to the module32 cards in this deck

Read all 32 cards as a list

Cost and latency optimization

Front
Sort these levers by what they improve: prompt caching, model routing, a provider batch API.
Back
Prompt caching cuts input cost and time to first token. Model routing cuts cost and often latency, at a quality risk. A batch API cuts cost only, in exchange for delayed results.
Front
Does streaming reduce total latency or cost?
Back
No. The first token reaches the user earlier, so it feels faster, but total generation time and the tokens billed are unchanged.
Front
How does provider prompt caching work, and how do you get cache hits?
Back
The provider reuses the computed state for an exact, identical prompt prefix, and reads cost a fraction of normal input price. Put static content (system prompt, tools, examples) first and variable content last. Output is unaffected.
Front
What is the risk of a semantic cache and how do you limit it?
Back
It can return an answer for a similar but different question, or across tenant or permission boundaries. Use a strict tuned threshold, scope keys by tenant and document version, set short TTLs and sample hits for review.
Front
Model routing or cascade: what is the saving and what must you check?
Back
Easy requests go to a small cheap model and hard ones escalate. Cost falls only if few requests escalate. Check quality per query type on a labelled set, because an unchanged average can hide a failing slice.
Front
Write the cost and latency formulas for one LLM request.
Back
Cost = input tokens x input price + output tokens x output price (plus cache terms). Latency = queueing + time to first token + output tokens / output speed. Measure which term dominates; it differs by workload.
Front
What do you measure before optimizing a slow or expensive LLM feature?
Back
Input and output tokens, time to first token, total time at p50 and p95, and cost per request, per stage and per tenant. Then optimize the dominant term.
Front
Why is max_tokens a weak way to make answers shorter?
Back
It cuts generation wherever the limit falls, even mid-sentence. Ask for a shorter format in the prompt (sentence or paragraph limits) and keep max_tokens as a safety ceiling.

LLM inference and serving

Front
Slow first token versus slow streaming: which metric and which cause?
Back
Slow first token is high TTFT, caused by queueing or long-prompt prefill. Slow streaming after a good start is high inter-token latency, caused by decode speed, memory bandwidth or batch pressure.
Front
The KV pool is full: what does the server do, and what does it cost?
Back
New requests queue, or the engine (vLLM, for example) preempts running ones and recomputes them later, which hurts end-to-end latency. Fixes: fewer KV-heads per request (GQA), a shorter context limit, KV quantization, more GPUs or replicas, or a lower concurrency cap.
Front
What do PagedAttention and continuous batching each do?
Back
PagedAttention stores the KV-cache in on-demand blocks to remove fragmentation, which allows larger batches and prefix sharing. Continuous batching schedules per decode step, so finished requests leave and new ones join immediately.
Front
Does speculative decoding change the output, and when does it help?
Back
No. The large model verifies the draft's tokens, so the output distribution is preserved (up to hardware numerics). It helps most at low to medium load, depends on the draft's acceptance rate, and adds a second component.
Front
What trade does chunked prefill make?
Back
It splits a long prompt so other users' decode steps keep running, which smooths their streaming. The cost is a somewhat slower first token for the long prompt. A smaller per-step token budget favors inter-token latency, a larger one favors TTFT.
Front
What does quantization buy and what does it cost?
Back
Fewer bytes per weight, so less memory and less to read per decode step (a 7B model is about 14 GB in 16-bit, about 7 GB in 8-bit). The cost is rounding error and hardware or kernel dependence, so test on your own eval set.
Front
Why can a self-hosted GPU cost more per token than an API?
Back
You pay for GPU hours whether or not they are busy, so cost per token is the hourly price divided by the tokens actually served. Low utilization raises it. Add redundancy and engineering time before comparing.
Front
What is goodput and why prefer it to throughput?
Back
A common definition is the rate of requests that meet the latency targets (for example TTFT and inter-token limits). DistServe uses a stricter variant: the highest rate at which a target share, such as 90 percent, meets both limits. Raw throughput can look high while many users get an unacceptable experience.

Observability and eval pipelines

Front
Why do LLM systems need an offline eval pipeline?
Back
Output varies and cannot be asserted with equality, and many parts change independently. A golden set run on every change catches regressions before users meet them.
Front
What do you record for each production LLM request?
Back
Trace id, prompt and model versions, inputs and outputs (privacy-safe), retrieved ids and scores, tokens, latency per stage, cost, outcome signals and errors.
Front
How do production failures feed back into the eval set?
Back
Sample traces (weighted toward thumbs-down, abstentions and escalations), label them, add them to the golden set with a slice tag after redaction, then gate future changes on the set.
Front
Why read golden-set results per slice?
Back
An average can rise while one slice collapses. Small sets are also noisy: with 100 cases near 90 percent the sampling noise is about 3 points, so one point is not a signal.
Front
What does online monitoring catch that offline eval misses, and the reverse?
Back
Online catches live outages, cost and latency shifts and drift in real traffic. Offline catches quality regressions, including rare intents, before release. Neither replaces the other.
Front
What privacy rules apply to LLM traces?
Back
They hold user text and company documents. Redact personal data and secrets, restrict access to full prompts, set retention and deletion, and apply the same rules before reusing traces in an eval set.
Front
Which alerts help for an LLM service and what are they compared with?
Back
Error and timeout rate, p95 latency per stage, cost per request, input token growth, abstention and empty-retrieval rate, feedback and sampled quality scores. Compare each with your own baseline, especially after deploys.

Production deployment of RAG

Front
What makes RAG ingestion safe to operate?
Back
It is idempotent (content-hash ids), incremental, propagates deletes with tombstones, and records the embedding model and chunker version on every chunk.
Front
How do you re-embed a corpus with zero downtime?
Back
Build a new index version in the background with the new model, run the eval gate, then swap the alias together with the query-side model. Keep the old version for a one-step rollback.
Front
How do you size a vector index?
Back
Raw size is vectors x dimensions x bytes per value (10M x 768 x 4 B is about 31 GB) before index overhead. That decides RAM versus disk, quantization and sharding. Shard for size, replicate for query rate.
Front
Why can output guardrails conflict with streaming?
Back
Streamed text is shown before it can be fully checked. Choose check-first (slower first token), stream-first (risk of showing unchecked text) or check in chunks.

RAG security, reliability and release

Front
Where do you enforce document permissions in RAG?
Back
At retrieval: store ACLs as metadata, filter by user or group at query time, and verify on the final context. Never rely on the prompt to hide data, and key caches on tenant, permissions and document version.
Front
Reranker is slow or down: what do you degrade first?
Back
Skip the rerank and serve hybrid retrieval results, a small known quality loss. Do not drop retrieval or permission filtering. If retrieval is empty or weak, abstain or ask a clarifying question.
Front
Order the release path for a RAG change (chunker, embedder, prompt, model).
Back
Offline eval gate on quality, latency and cost, then shadow traffic, then a small canary, then full rollout, with rollback through an alias or feature flag. Version every component.
Front
A user asks for their data to be deleted. Where must the deletion reach in a RAG system?
Back
Vectors, caches and logs, not only the source store. Check your legal requirements.
Front
Which parts of a RAG system do you version for release?
Back
Everything that can change behaviour: prompts, embedding model, chunker, index, reranker and LLM. Put changes behind feature flags so rollback is one switch.