Making attention cheaper

Most of a transformer costs the same per token whether the context holds 100 tokens or 100,000. The matrix multiplies in the MLP and the projections see one token at a time. Attention is different: every token looks at every earlier token, so its work and its memory grow with context length. Computed from scratch over a prompt, attention is quadratic in sequence length. With a KV cache, each decode step is linear in the length of the context so far.

That growth shows up as four separate problems, and the techniques you will meet in every serving engine each fix one of them. FlashAttention avoids building the score matrix. Grouped-query attention, latent attention and sliding windows shrink the KV cache. PagedAttention stops the KV cache from wasting memory on reservations. Chunked prefill stops one long prompt from freezing everyone else's stream.