Glossary
Decode
The phase after prefill where one token is generated per step. Each step has to read the weights and the KV cache, so it is usually limited by memory bandwidth.
Where it is explained
See also
Terms explained in the same lessons.
Glossary
The phase after prefill where one token is generated per step. Each step has to read the weights and the KV cache, so it is usually limited by memory bandwidth.
Terms explained in the same lessons.