Cheat sheet: Production: cost, serving and observability

Formulas: cost = input tokens x input price + output tokens x output price (plus cache terms). Latency = queueing + time to first token + output tokens / output speed.

Cost levers (each has a quality or freshness risk): provider prompt caching (stable prefix first), shorter prompts and fewer chunks, concise output format with max_tokens as a ceiling, model routing or cascade, exact and semantic response caches (scope by tenant and version), quantized or distilled models, provider batch API (cheaper, delayed, offline only).

Latency levers: shorter output, smaller or faster model, prompt caching (prefill), parallelize independent calls. Streaming improves perceived latency only: not cost, not total time.

Observability: trace every request (versions, retrieved ids and scores, tokens, latency per stage, cost, outcome signals), redact and expire traces. Live monitoring catches outages, cost and drift. A golden set run on every change catches regressions, read per slice.

Quick answers#

• Measure first: tokens in and out, TTFT, p95, cost per request, per stage. Optimize the dominant term.
• Safe change: offline eval, then shadow, canary, rollback via alias or flag.
• Serving: prefill is compute-heavy (sets TTFT); decode is memory-bandwidth-bound (sets inter-token latency). KV-cache memory caps concurrency (formula in How LLMs work). Continuous batching and PagedAttention raise throughput; batching can raise per-request latency under load. FlashAttention is exact. Speculative decoding keeps the output distribution. Quantization trades quality for memory and speed.
• Self-host or API: cost per token = hourly price / tokens served, so utilization decides.
• RAG in production: idempotent, versioned ingestion with delete propagation; re-embed into a new index and swap an alias; permissions enforced at retrieval; cache keys include tenant, permissions and document version.