Lesson deck
LLM inference and serving flashcards
Reviewing needs JavaScript. You can read every card in the list below.
Loading your cards...
Back to the lesson8 cards in this deck
Read all 8 cards as a list
- Front
- Slow first token versus slow streaming: which metric and which cause?
- Back
- Slow first token is high TTFT, caused by queueing or long-prompt prefill. Slow streaming after a good start is high inter-token latency, caused by decode speed, memory bandwidth or batch pressure.
- Front
- The KV pool is full: what does the server do, and what does it cost?
- Back
- New requests queue, or the engine (vLLM, for example) preempts running ones and recomputes them later, which hurts end-to-end latency. Fixes: fewer KV-heads per request (GQA), a shorter context limit, KV quantization, more GPUs or replicas, or a lower concurrency cap.
- Front
- What do PagedAttention and continuous batching each do?
- Back
- PagedAttention stores the KV-cache in on-demand blocks to remove fragmentation, which allows larger batches and prefix sharing. Continuous batching schedules per decode step, so finished requests leave and new ones join immediately.
- Front
- Does speculative decoding change the output, and when does it help?
- Back
- No. The large model verifies the draft's tokens, so the output distribution is preserved (up to hardware numerics). It helps most at low to medium load, depends on the draft's acceptance rate, and adds a second component.
- Front
- What trade does chunked prefill make?
- Back
- It splits a long prompt so other users' decode steps keep running, which smooths their streaming. The cost is a somewhat slower first token for the long prompt. A smaller per-step token budget favors inter-token latency, a larger one favors TTFT.
- Front
- What does quantization buy and what does it cost?
- Back
- Fewer bytes per weight, so less memory and less to read per decode step (a 7B model is about 14 GB in 16-bit, about 7 GB in 8-bit). The cost is rounding error and hardware or kernel dependence, so test on your own eval set.
- Front
- Why can a self-hosted GPU cost more per token than an API?
- Back
- You pay for GPU hours whether or not they are busy, so cost per token is the hourly price divided by the tokens actually served. Low utilization raises it. Add redundancy and engineering time before comparing.
- Front
- What is goodput and why prefer it to throughput?
- Back
- A common definition is the rate of requests that meet the latency targets (for example TTFT and inter-token limits). DistServe uses a stricter variant: the highest rate at which a target share, such as 90 percent, meets both limits. Raw throughput can look high while many users get an unacceptable experience.