Disaggregated prefill and decode

Prefill and decode are different jobs sharing one GPU. Prefill crunches thousands of prompt tokens in one burst and is limited by compute. Decode produces one token per step for every running request and is limited by memory bandwidth. When both run in the same batch, a big prefill makes the step longer for every user who is waiting on their next token.

Disaggregation puts them on different GPUs. A prefill worker computes the prompt's KV cache and the first token, ships the KV cache to a decode worker, and the decode worker streams the rest. Each pool can then be sized and tuned for one metric: prefill for time to first token, decode for inter-token latency. The price is a transfer of the KV cache for every request, and a more complicated system.