Continuous batching

Decode at small batch sizes wastes a GPU. Each step reads every weight from memory to produce one token, and the arithmetic cores sit mostly idle while that read happens. Put more sequences in the same step and the weights are read once for all of them, so the extra tokens come almost for free. Batching is how a serving engine turns spare compute into throughput.

The catch is that LLM requests do not finish together. One user wants a ten-token answer, another a thousand-token explanation. If the batch has to wait for its longest member, most of its slots spend most of their time producing nothing. Continuous batching, also called in-flight batching, fixes that by rebuilding the batch at every decode step: finished sequences leave, waiting ones join, and the GPU never idles a slot while there is work in the queue.