Cost and latency optimization

Story

Hide story

Past the second tunnel, Marta Voss slaps a fuel receipt on the counter. The storm has doubled the bill, and she wants it halved by Friday. Her thermos is empty.

Everyone has a remedy. Fit a smaller engine. Dim half the lamps. Ask the announcer to talk less. Nobody has read a gauge yet.

What would you do first?

Concept#

Two numbers decide whether an LLM feature survives contact with real traffic: what one request costs, and how long the user waits.

Cost is usually metered per token. A request costs input tokens x input price + output tokens x output price, plus smaller terms if you use prompt cachingPrompt cachingA provider feature that reuses the computed state of an identical prompt prefix. Cache reads cost a fraction of normal input and the first token comes sooner, while cache writes can cost a premium. Put static content first. Not the same as prefix caching, which happens inside your own serving engine.Open the glossary entryClose. Providers commonly list output tokens at a higher per-token price than input tokens, so a short answer can still be a large share of the bill. Some providers also add price tiers above a long-context threshold, so read the price page of the model you actually use. Billing is per token, not quadratic in length: the quadratic attention work from How LLMs work is hidden inside the provider’s price, and it shows up as slower prefill on very long prompts. It only becomes your own bill when you run the GPUs yourself (see LLM inference and serving).

Latency has two parts: the wait before the first token, then the time to write the rest.

total latency = queueing + time to first token (TTFT) + output tokens / output speed

TTFTTTFTShort for time to first token: the delay from sending a request until the first output token arrives.Open the glossary entryClose grows with prompt length and load, because the whole prompt is processed first (prefill). Output speed is tokens per second during decode. If the model also generates reasoning tokens before the visible answer, those usually count as output tokens for both cost and time, so check the usage fields your provider returns.

Why It Matters#

“It is too slow” and “it is too expensive” are two frequent follow-ups after a GenAI design. Both have a cheap wrong answer (“use a smaller model”) and a better one (measure, find the dominant term, pull the lever that attacks it, and check what it did to quality).

Every lever below moves a trade-off. If you cannot say what a lever costs in quality, freshness or complexity, you have not understood it yet.

Mental Model#

Think of a taxi meter and a stopwatch running on the same journey. The meter charges per token read and, at a higher rate, per token written. The stopwatch runs for a fixed wait while the driver reads your address (TTFT), then for as long as it takes to say the route aloud, word by word (output tokens divided by speaking speed).

So there are three dials: read less, write less, or pay less per word. Streaming is a fourth thing that changes only what the passenger feels, not the meter or the stopwatch.

Measure first#

Do not optimize on a hunch. Tag every request with these numbers, per stage if the request has several (retrieve, rerank, generate):

• input tokens and output tokens (and cached tokens, if your provider reports them);
• TTFT and total time, as p50 and p95Percentile latency (p50, p95)The time under which 50 percent or 95 percent of requests finish. The tail (p95) shows what the slower users experience, which averages hide.Open the glossary entryClose (the slowest 5 percent are what users complain about);
• cost per request, and cost per tenant or per feature;
• which model, prompt version and cache outcome handled the request.

Then ask which term dominates. A long-prompt RAG app is often dominated by input tokens for cost, and by TTFT when answers are short. A chat or report writer is often dominated by output tokens. An agent loop is dominated by the number of model calls. The right lever depends on the answer, and a lever aimed at the wrong term does nothing.

One more rule from the provider docs: first get a prompt that works well without constraints, then optimize. Squeezing latency too early hides what top quality looks like (Anthropic, reducing latency, accessed 2026-10-09).

Worked example (illustrative numbers)#

All figures here are illustrative, not any provider’s prices. Assume input costs 2 dollars per million tokens and output costs 10 dollars per million.

One request has 3,000 input tokens and 400 output tokens.

• Input: 3,000 x 2 / 1,000,000 = $0.006.
• Output: 400 x 10 / 1,000,000 = $0.004.
• Total: $0.010 per request, or $1,000 per day at 100,000 requests.

Output is only 400 of 3,400 tokens (about 12 percent) but 40 percent of the cost. That is why a “be concise” instruction is a cost lever.

Now latency. Assume TTFT 0.6 s and output speed 50 tokens/s. Total = 0.6 + 400 / 50 = 8.6 s. Almost all of it is generation. Halving the prompt might shave a fraction of the 0.6 s. Halving the answer to 200 tokens saves 4 s. Streaming would show the first word at 0.6 s, but the answer still finishes at 8.6 s.

Finally, prompt caching. Suppose 2,500 of the 3,000 input tokens are a stable prefix, and (as a typical documented pattern) cache reads cost 0.1x the input price and cache writes 1.25x.

• Every request hits: prefix = 2,500 x 0.2 / 1,000,000 = $0.0005 instead of $0.005. Input becomes 0.0005 + 500 x 2 / 1,000,000 (= 0.001) = $0.0015. Total $0.0055, down 45 percent.
• Only 80 percent hit, and each miss rewrites the cache at 1.25x: prefix = 0.8 x 0.0005 + 0.2 x (2,500 x 2.5 / 1,000,000 = 0.00625) = 0.0004 + 0.00125 = $0.00165. Input $0.00265, total $0.00665, down about 33.5 percent.

The hit rate, not the headline discount, decides the saving.

The levers and what each costs you#

Notice that no lever is free, and that two of them (batch API, streaming) do not touch latency in the way people assume.

Provider prompt caching#

A transformer must process the whole prompt before it generates. If the first 2,500 tokens are identical on every request (system prompt, tool definitions, few-shot examples, a long reference document), the provider can keep the computed state for that prefix and skip the work next time. This is the KV-cache idea from How LLMs work, applied across requests.

What the Anthropic documentation says, as an example of the pattern (accessed 2026-10-09):

• The cache matches an exact prefix, in the order tools, system, messages. A change anywhere in the prefix invalidates it from that point on.
• Prompt caching has no effect on output token generation.
• Reads cost a fraction of the base input price, writes cost a premium, the default lifetime is 5 minutes and is refreshed each time the entry is used, and there is a minimum cacheable length that varies by model.
• The usage fields report cache-read and cache-creation tokens, so you can compute your real hit rate.

Other providers and open-source engines such as vLLM (automatic prefix caching) work on the same principle, with different prices, lifetimes and rules. Learn the principle, then read your provider’s current page.

The practical rule: static content first, variable content last. A timestamp, user name or request id at the top of the system prompt silently turns every request into a miss.

Do not confuse this with a response cache. Prompt caching still runs the model and generates a fresh answer. A response cache skips the model entirely.

Response caches#

• Exact cache: key on the full normalized input plus everything that changes the answer (model, prompt version, retrieval corpus version, tenant). Safe, but only repeated identical questions hit.
• Semantic cacheSemantic cacheA cache that returns a stored answer for a similar question. It can serve a wrong answer or cross a tenant boundary, so use a strict threshold and scope keys.Open the glossary entryClose: embed the question (Embeddings and similarity search) and reuse a stored answer when a past question is close enough. The risk is that “close in embedding space” is not “same question”: “cancel my order” and “cancel my subscription” can be near neighbours with different answers. Mitigate with a strict threshold tuned on real pairs, scoping by tenant and permissions, short TTLs, invalidation when source documents change, and sampling hits for review.

Streaming, batching and output limits#

• StreamingStreamingSending output tokens as they are generated. It improves perceived latency, not total latency or cost.Open the glossary entryClose returns tokens as they are generated. TTFT is what the user feels, so it improves perceived latency. Total generation time and cost are unchanged. It is not a cost lever.
• Batching means two different things. A provider batch APIBatch APIA provider interface that runs requests asynchronously at lower cost in exchange for delayed results.Open the glossary entryClose gives a discount (for Anthropic’s Message Batches API, 50 percent of standard prices, with most batches finishing within an hour and a 24-hour expiry, accessed 2026-10-09) in exchange for asynchronous delivery. It suits evals, backfills and nightly jobs, not a chat box. Batching on your own GPUs (continuous batching) raises throughput per GPU and so lowers cost per token, but each request can wait longer when the server is busy. See LLM inference and serving.
• Output limits: max_tokens is a safety net, not a style tool. It cuts the answer wherever the limit falls. Ask for a shorter format in the prompt (sentence or paragraph limits work better than word counts), then keep max_tokens as the ceiling.

Model routing#

Send easy requests to a small, cheap model and escalate hard ones. Two shapes: a router (classify first, then call one model) and a cascade (try the small model, escalate when its answer looks weak).

Illustrative arithmetic. Let the large model cost 1 unit per request and the small one 0.1. If 70 percent of traffic is easy and a perfect free router sends it to the small model: 0.7 x 0.1 + 0.3 x 1 = 0.37, a 63 percent saving. With a cascade, hard requests pay for both attempts: 0.7 x 0.1 + 0.3 x (0.1 + 1) = 0.07 + 0.33 = 0.40, a 60 percent saving. Real savings are lower once you add the router’s own cost and the requests it sends the wrong way, and the price is quality: you must measure the quality of the routed system on a labelled set, per query type, before and after (Evaluating LLM systems).

Diagram#

Diagram: A request first checks the response cache and returns immediately on a hit.

Real-World Application#

A support assistant has a 4,000-token system prompt with policies and tool definitions, plus a user question. Traces show input tokens dominate cost and TTFT. The team moves the per-user details to the end of the prompt so the 4,000-token prefix is identical across requests, turns on provider prompt caching, and watches the cache-read share in the usage fields. They route FAQ-style questions to a small model after confirming on a labelled set that quality holds, ask for answers in at most three sentences, and stream the answer so the first words appear quickly. Each change is measured against the baseline, one at a time.

Interview Perspective#

Expect: “your RAG app is too slow or too expensive, what do you do?” and “explain semantic caching and its risks”. A strong answer starts with measurement (“which stage and which term dominates?”), names several levers with their risk, and separates cost from perceived latency. The weak answer is a single lever (“use a smaller model”) or “turn on streaming” as a cost fix.

Provenance: Common industry pattern. These are standard GenAI system-design follow-ups, not a verified list for any named company.

Practice Questions#

• Basic: Name three levers that reduce cost and say which of them do not reduce real latency.
• Intermediate: Why does streaming not reduce total latency, and why does it still feel faster?
• Intermediate: Your prompt starts with the current date and time. What does that do to prompt caching, and how do you fix it?
• Advanced: A semantic cache returns a wrong answer for a near-duplicate question. How do you detect it and limit the damage?
• Product-Company: Design a routing strategy for a coding assistant that handles both “fix this typo” and “refactor this module”. How do you evaluate it?

Activity#

For a hypothetical support bot, write down five example queries and decide, with a one-line reason each, whether the small or the large model should answer. Then list the signal your router would use for each one (length, keywords, retrieval score, a classifier) and what you would measure to find out you were wrong.

Homework#

Take this worked example (3,000 input tokens, 400 output tokens, 100,000 requests per day, the illustrative prices above). Compute the daily cost of three plans with your arithmetic shown: (a) baseline, (b) prompt caching at an 80 percent hit rate on a 2,500-token prefix, (c) routing 60 percent of traffic to a model that costs one tenth. State two assumptions you made and the quality check you would run before shipping each plan.

Then design a caching strategy (exact plus semantic) for a FAQ chatbot, including how you would invalidate stale answers when the underlying documents change.

Quick Revision#

• Cost = input tokens x input price + output tokens x output price. Latency = queueing + TTFT + output tokens / output speed.
• Measure by stage first (tokens, TTFT, p95, cost per request). Optimize the dominant term.
• Prompt caching needs a stable prefix and a high hit rate. It does not change the output.
• Streaming changes perceived latency only. A provider batch API trades delay for a discount.
• Semantic caching can answer a similar but different question. Scope it and invalidate it.
• Routing saves money only if you measure the quality of the routed system.

Check yourself#

• Walk-through: Your LLM feature has p95 8 s and costs three times its budget. List the measurements you take first, then five levers with the quality cost of each.
• Variation: Design a model router. What signals pick the model, and how do you evaluate that it does not degrade quality?
• Common misconception: Optimizing before measuring; treating streaming or batching as a universal cost lever; believing every lever is quality-neutral.

Related#

• How LLMs work
• LLM inference and serving
• Observability and eval pipelines