KV Cache

KV Cache caches already computed key-value pairs (K and V of the attention mechanism) of previous tokens to avoid recalculating them at each generation step. The cache works by prefix: if even one token at the beginning is changed, the entire cache of all layers is invalidated (output of the 1st layer is input to the 2nd, and so on). Decoding still iterates through all cached K and V, so a long context slows down generation linearly.

Related: [Three Rules of KV Cache Friendliness], Prompt Cache, Chat Template