KV cache tiers and cache-aware routing
KV cache tiers and cache-aware routing
Prefix caching on one GPU has two limits. GPU memory holds only a few dozen long prefixes before running requests push them out, and a production fleet has many replicas, each with its own separate cache. A follow-up question that lands on the wrong replica, or arrives after its prefix was evicted, pays for full prefill again.
Two techniques remove those limits. Tiered KV storage moves evicted blocks down to CPU memory, local NVMe and network storage instead of discarding them, and loads them back when needed. Cache-aware routing sends each request to the replica that already holds its prefix. Together they turn a per-GPU optimization into a fleet-wide one.
Continue reading
Pro unlocks every video lesson, the full notes, worked sizing and cost calculations, and runnable configs across the Inference Engineering course, from GPUs and kernels to quantization, speculative decoding, disaggregation and production autoscaling.