Lesson deck
Cost and latency optimization flashcards
Reviewing needs JavaScript. You can read every card in the list below.
Loading your cards...
Back to the lesson8 cards in this deck
Read all 8 cards as a list
- Front
- Sort these levers by what they improve: prompt caching, model routing, a provider batch API.
- Back
- Prompt caching cuts input cost and time to first token. Model routing cuts cost and often latency, at a quality risk. A batch API cuts cost only, in exchange for delayed results.
- Front
- Does streaming reduce total latency or cost?
- Back
- No. The first token reaches the user earlier, so it feels faster, but total generation time and the tokens billed are unchanged.
- Front
- How does provider prompt caching work, and how do you get cache hits?
- Back
- The provider reuses the computed state for an exact, identical prompt prefix, and reads cost a fraction of normal input price. Put static content (system prompt, tools, examples) first and variable content last. Output is unaffected.
- Front
- What is the risk of a semantic cache and how do you limit it?
- Back
- It can return an answer for a similar but different question, or across tenant or permission boundaries. Use a strict tuned threshold, scope keys by tenant and document version, set short TTLs and sample hits for review.
- Front
- Model routing or cascade: what is the saving and what must you check?
- Back
- Easy requests go to a small cheap model and hard ones escalate. Cost falls only if few requests escalate. Check quality per query type on a labelled set, because an unchanged average can hide a failing slice.
- Front
- Write the cost and latency formulas for one LLM request.
- Back
- Cost = input tokens x input price + output tokens x output price (plus cache terms). Latency = queueing + time to first token + output tokens / output speed. Measure which term dominates; it differs by workload.
- Front
- What do you measure before optimizing a slow or expensive LLM feature?
- Back
- Input and output tokens, time to first token, total time at p50 and p95, and cost per request, per stage and per tenant. Then optimize the dominant term.
- Front
- Why is max_tokens a weak way to make answers shorter?
- Back
- It cuts generation wherever the limit falls, even mid-sentence. Ask for a shorter format in the prompt (sentence or paragraph limits) and keep max_tokens as a safety ceiling.