Prefix caching

Prefill turns every prompt token into keys and values for every layer. For a given model, those keys and values depend only on the tokens up to that position. So if two requests begin with the same 5,000 tokens, the KV cache for those 5,000 tokens is identical, and computing it twice is waste. Prefix caching keeps KV cache blocks after a request finishes and hands them to the next request whose prompt starts the same way.

The rule that follows is strict: the match is on an exact token prefix. One different token at position 40 means everything from position 40 onward must be recomputed, even if the remaining 4,960 tokens are byte-for-byte the same. Prompt layout, not model choice, decides whether you get the benefit.