Inter-token latency

The gap between successive output tokens (also called time per output token, TPOT), which sets how smooth streaming feels. It reflects decode speed and batch pressure.

All terms, A to Z