LLM inference and serving

Story

Hide story

Midway up the pass, the queue for announcements outside the Voice’s door grows faster than it shrinks. Latency the cat wakes, crosses the carriage, and is still ahead of the slowest waiter.

Marta checks the gauges. The engine is not working hard, and the line keeps growing.

If the engine is not the limit, what is the queue waiting for?

Concept#

Serving an LLM is running the generation loop from How LLMs work for many users at once on expensive GPUs. That lesson explained prefill (read the whole prompt in parallel, compute-heavy) and decode (one token per step, mostly waiting on memory). This lesson is about the consequences: how a server schedules, stores and speeds up those two phases, and what each trick costs you.

Two scarce resources shape everything. Memory bandwidth limits how fast decode can go. Memory capacity limits how many requests can be in flight, because every active request holds a KV-cacheKV cacheStored key and value vectors of earlier tokens, reused at each decode step so they are not recomputed. It saves compute but its memory grows with context length and concurrent requests. It is not a response cache.Open the glossary entryClose.

Why It Matters#

“Why is our LLM slow or expensive, and how would you serve it at scale?” is a standard GenAI-depth and system-design follow-up. It separates people who only call APIs from people who can reason about GPUs, batching and memory. Even if you never self-host, the same ideas explain provider behaviour: why long prompts delay the first token, why long answers take so long to finish, and why prompt caching exists.

Mental Model#

A GPU is a very fast calculator with a narrow straw to its memory. Generating one token for one user barely uses the calculator but drains the straw, because the straw has to carry all the model weights past the calculator for every single token.

Batching is a bus. One trip through the weights can serve many riders, so the cost of the trip is shared. Each rider also carries a bag (their KV-cache) that takes a seat’s worth of memory, so the bus fills up from the bags before it fills up from the riders. A good scheduler lets riders get on and off at every stop rather than waiting for the whole bus to finish its route.

Why decode is bandwidth-bound (illustrative arithmetic)#

A 7-billion-parameter model in 16-bit numbers has weights of about 7e9 x 2 bytes = 14 GB. For one request, each decode step reads those 14 GB. Suppose, illustratively, that the GPU can move 2,000 GB per second. Then one stream cannot go faster than 2,000 / 14, about 143 tokens per second, however much arithmetic power the chip has. (This ignores reading the KV-cache and other overheads, so real speeds are lower.)

Two consequences follow, and they explain most of the techniques below.

• Make the weights smaller and the bound rises. Weights in 8-bit numbers are about 7 GB, which lifts the same bound to roughly 286 tokens per second.
• Serve many requests per weight read and each trip is shared. This is what batching does.

KV-cache memory is the concurrency limit#

The formula is in How LLMs work: 2 x layers x KV-heads x head-dim x sequence length x batch x bytes per value. Here is what it means for a server.

Illustrative capacity plan: an 80 GiB GPU, 90 percent of it usable (72 GiB), a 7B-class model in 16-bit numbers (about 13 GiB of weights), 32 layers, 128-dimension heads, 4,096-token requests.

• With 32 KV-heads: 2 x 32 x 32 x 128 x 2 = 524,288 bytes per token (0.5 MiB), so a 4,096-token request holds 2 GiB. The KV pool is 72 - 13 = 59 GiB, which fits about 29 requests.
• With grouped-query attention and 8 KV-heads, the per-token cost is 4 times smaller: 0.5 GiB per request, so about 118 requests.

Same GPU, same weights, four times the concurrency, from one architecture choice. When the pool is full, new requests queue, and engines such as vLLM may pause and later recompute running ones, which hurts end-to-end latency (vLLM optimization guide, accessed 2026-10-09). So “how many users fit on one GPU” is mostly a KV-cache question.

Metrics that tell you where it hurts#

• Time to first token (TTFT): queueing plus prefill. Grows with prompt length and with load.
• Inter-token latency (ITL, also called time per output token): how fast decode runs once started. This is what makes streaming feel smooth or choppy.
• Throughput: tokens per second across all users. A cost metric.
• GoodputGoodputThe rate of requests that meet the latency targets. Throughput can look high while many users get an unacceptable experience. DistServe uses a stricter version: the highest request rate at which a target share of requests meets the limits.Open the glossary entryClose: a common definition is the rate of requests that meet the latency targets you promised (for example TTFT and ITL limits), so it counts only useful work. DistServe uses a stricter variant: the highest request rate at which a target share (for example 90 percent) of requests meet both limits (Zhong et al., 2024).

Symptoms map to causes. Slow first token only at peak hours points to queueing or long prompts. Slow streaming after a good first token points to decode speed, memory bandwidth or batch pressure. High cost with fine latency points to poor batching or low utilization.

Techniques and what each costs#

Continuous batching#

Static batching holds a group of requests until every one of them finishes, so short answers sit idle behind the longest. Continuous (in-flight) batchingContinuous batchingScheduling work per decode step so finished requests leave a batch and new ones join immediately.Open the glossary entryClose decides batch membership at every decode step: finished requests leave and queued ones join. The Anyscale post that popularised it reported up to 23x throughput versus naive static batching using vLLM on an A100 with OPT-13B in 2023, of which continuous batching alone gave roughly 8x (Anyscale, 2023-06-22). Treat that as one benchmark in one setting, not a number to quote for your workload.

PagedAttention and prefix caching#

If you reserve contiguous memory for each request’s maximum length, most of it sits empty and the batch stays small. PagedAttentionPagedAttentionStoring the KV cache in on-demand blocks to avoid fragmentation, which allows larger batches and prefix sharing.Open the glossary entryClose stores the KV-cache in fixed-size blocks allocated on demand, like pages in an operating system. The paper reports that vLLM improved throughput by 2 to 4 times over earlier systems at the same latency (Kwon et al., 2023). Because blocks are addressable, identical prompt prefixes can share them: automatic prefix caching hashes full blocks and reuses a matching prefix, reducing prefill work without changing outputs (vLLM docs, accessed 2026-10-09). This is the self-hosted twin of the provider prompt caching in Cost and latency optimization. It speeds up prefill, not decode.

Chunked prefill#

A 30,000-token prompt can occupy the GPU for a long prefill while other users’ decode steps wait, so their streams stutter. Chunked prefillChunked prefillSplitting a long prompt's prefill into pieces so other users' decode steps keep running. It smooths streaming for others at some cost to the long prompt's first token.Open the glossary entryClose splits the prompt and mixes chunks with decode work. The vLLM guide describes the knob as a trade: a smaller per-step token budget gives better inter-token latency, a larger one gives better time to first token (vLLM optimization guide, accessed 2026-10-09).

FlashAttention#

Standard attention writes a large n x n score matrix to the GPU’s main memory and reads it back. FlashAttentionFlashAttentionA way of computing attention that reduces reads and writes to slow GPU memory, giving the same result faster.Open the glossary entryClose computes the same result in tiles that stay in fast on-chip memory, cutting memory traffic. It is exact, not an approximation (Dao et al., 2022).

Quantization#

Store weights (and sometimes activations or the KV-cache) in 8-bit or 4-bit formats. A 7B model’s weights go from about 14 GB to about 7 GB and about 3.5 GB, plus some overhead for scaling factors, which shrinks the bytes read per decode step and frees memory for KV-cache. The cost is rounding and clipping error (NVIDIA, 2023-11-17). Weights are easier to quantize than activations, which have outliers. The speed-up depends on the kernels and hardware you run, so choose the format by testing on your own eval set, not by default.

Speculative decoding#

A cheap draft process proposes several tokens and the large model verifies them in one pass, keeping the matching prefix. Done correctly, the output distribution is unchanged: the original paper reports 2x to 3x faster decoding with identical outputs on T5X (Leviathan et al., 2022). The vLLM documentation says it is theoretically lossless up to hardware numeric precision, aimed at medium-to-low load and memory-bound workloads, with, for most methods, smaller gains at high load, and recommends measuring on your own traffic (vLLM docs, accessed 2026-10-09). Speed-up depends on how often the draft is right.

Parallelism and disaggregation#

Split a model that does not fit on one GPU with tensor parallelism (needs fast links between GPUs) or pipeline parallelism, and add data-parallel replicas for more requests per second. Disaggregation runs prefill and decode on separate pools because their bottlenecks differ; DistServe reports serving up to 7.4x more requests, or meeting up to 12.6x tighter latency targets, than the systems it compared against, in its own tests (Zhong et al., 2024).

Diagram#

Diagram: Requests enter a scheduler that batches them continuously, run prefill, store their keys and values in a paged cache with shared prefixes,...

Diagram: A triage tree: check time to first token first and fix queueing or long prompts, then inter-token latency and fix memory bandwidth, then...

Self-host or use an API?#

A hosted API sells you tokens and hides everything above. Self-hosting sells you GPU hours and leaves the engineering to you. The deciding quantity is utilization.

Illustrative break-even: a GPU costs 2 dollars per hour (invented price) and your engine sustains 1,000 output tokens per second across users (invented throughput). That is 3,600,000 tokens an hour, so 2 / 3.6 = about $0.56 per million tokens at full load. At 30 percent utilization the same GPU gives 1,080,000 tokens an hour, so the price is 2 / 1.08 = about $1.85 per million. An idle GPU still costs money, an API does not. This arithmetic also leaves out redundancy for failover, engineers’ time, and the quality gap between an open model and a frontier one.

Reasons to self-host: steady high volume, strict data residency, the need to fine-tune or control the model, or latency control. Reasons not to: spiky or low traffic, a small team, or needing the strongest available model. A common hybrid is self-hosted for the bulk and an API fallback for overflow and hard cases.

Real-World Application#

Self-hosting an open model behind vLLM or a similar engine: pick quantization by eval, turn on prefix caching for a long shared system prompt, autoscale replicas on queue depth rather than CPU, set separate alerts on TTFT and ITL, and keep an API fallback for overflow.

Interview Perspective#

Expect: “prefill versus decode”, “what is the KV-cache and how big is it”, “how does continuous batching work”, “self-host or API for 50 million tokens a day?”. Mistakes: saying quantization is free, confusing a response cache with the KV-cache, claiming speculative decoding changes outputs, or assuming a bigger batch always lowers latency (it raises throughput; per-request latency can get worse).

Provenance: Common industry pattern. These are standard serving topics for ML infrastructure and GenAI roles, not a verified list for any named company.

Practice Questions#

• Basic: Why is decode memory-bandwidth-bound while prefill is compute-bound?
• Intermediate: Compute KV-cache per token for a model with L layers, H_kv KV-heads and head dimension d in 16-bit numbers. How does GQA change it, and what does that do to concurrency?
• Advanced: The latency target is violated only under load with long prompts. Diagnose and fix.
• Product-Company: Self-host or API for 50 million tokens a day with data-residency limits? Decide and justify with a break-even estimate.

Activity#

Compute KV-cache memory for a 4,096-token request on two architectures (32 KV-heads versus 8), then estimate how many concurrent requests fit on an 80 GiB GPU after the weights. State your assumptions (usable memory fraction, precision).

Homework#

Write a one-page serving plan: engine, quantization choice and how you validate its quality loss, batching, autoscaling signal, latency targets for TTFT and ITL, and fallback.

Quick Revision#

• Prefill sets TTFT. Decode sets inter-token latency and is memory-bandwidth-bound.
• KV-cache memory caps concurrency. GQA shrinks it. A full pool means queueing or preemption.
• Continuous batching and PagedAttention raise throughput; batching can raise per-request latency under load.
• Prefix caching and chunked prefill protect time to first token and smooth streaming.
• Quantization trades quality for memory and speed. Speculative decoding keeps the output distribution. Measure both on your own traffic.
• Self-hosting wins on utilization, loses on idle time and operations.

Check yourself#

• Walk-through: Users complain of a slow first token at peak hours, but streaming speed is fine once it starts. What do you measure and change?
• Variation: Streaming is choppy but TTFT is fine. What now?
• Common misconception: “A bigger batch always lowers latency.” It raises throughput; per-request latency can worsen.

Related#

• How LLMs work
• Cost and latency optimization
• Fine-tuning methods: SFT, LoRA, QLoRA and DPO