KV cache

Stored key and value vectors of earlier tokens, reused at each decode step so they are not recomputed. It saves compute but its memory grows with context length and concurrent requests. It is not a response cache.

All terms, A to Z