The KV cache and its memory bill
The KV cache and its memory bill
At each decode step, the new token's query has to be scored against the keys of every earlier token, and it collects their values. Recomputing those keys and values from scratch would mean rerunning the whole prompt through every layer for every generated token. So the engine keeps them. That store is the KV cache, and it turns decode from quadratic work into a read of saved tensors.
The trade is memory. The cache grows by a fixed number of bytes for every token of every request that is running, prompt and output alike. Weights are a one-time cost per GPU. The cache is a per-user, per-token cost, and it is usually the number that decides how many conversations fit.
Continue reading
Pro unlocks every video lesson, the full notes, worked sizing and cost calculations, and runnable configs across the Inference Engineering course, from GPUs and kernels to quantization, speculative decoding, disaggregation and production autoscaling.