KV Cache

KV Cache caches already computed key-value pairs (K and V of the attention mechanism) of previous tokens, to avoid recalculating them at each generation step.

The cache works by prefix: if even one token at the beginning is changed, the entire cache of all layers is invalidated (output of the 1st layer – input of the 2nd, and so on).

Decoding still iterates through all cached K and V, so a long context slows down generation linearly.