Production project
Serving and latency profile
Builds on 2 lessonsabout 1 h 5 min of reading first
In this project
What you need first
- Completed:Cost and latency optimizationProduction: cost, serving and observability
- Completed:LLM inference and servingProduction: cost, serving and observability
Skills you will practise
- Cost and latency
- Inference and serving
Problem
Measure where latency and throughput come from when serving a small model.
Requirements
- A small model served with a serving engine.
- TTFT, inter-token latency and throughput at several concurrency levels.
- At least one experiment with quantization or prefix caching.
Prerequisites
Tasks
- To do: Serve a small model with a serving engine.
- To do: Measure TTFT, inter-token latency and throughput at several concurrencies.
- To do: Try quantization and prefix caching, and measure again.
- To do: Write up the results.
Evaluation
Record real measurements and never invent numbers. The finished project should give you: latency and cost reasoning backed by real numbers.