Latency budgets and workload shapes
Latency budgets and workload shapes
A latency target like "tab completion within 300 ms" is not yet something you can engineer against. It becomes one when you split it into a budget: so many milliseconds for the network, so many for waiting in a queue, so many for prefill, and the rest for decode. Once the budget is written down, every proposed change has an obvious question attached: which line does it shrink, and by how much?
The budget depends on the workload's shape. How long is the input? How long is the output? Do requests arrive one at a time while someone waits, or all at once overnight? Two workloads served by the same model on the same GPU can have nothing else in common, and Kite has four of them.
Continue reading
Pro unlocks every video lesson, the full notes, worked sizing and cost calculations, and runnable configs across the Inference Engineering course, from GPUs and kernels to quantization, speculative decoding, disaggregation and production autoscaling.