Client code, streaming, and protocols
Client code, streaming, and protocols
Everything so far has made the server faster. The user, though, feels the whole path: network, queue, prefill, and every output token. Client code controls a surprising share of that path. It decides whether each request opens a new connection, whether the user reads the first sentence after half a second or waits ten seconds for the whole answer, and whether a request the user no longer wants keeps occupying a batch slot.
Three client habits pay for themselves on every workload: keep connections open, stream what a person is reading, and cancel what nobody is waiting for. The fourth decision is the protocol, and it follows from who is on the other end.
Continue reading
Pro unlocks every video lesson, the full notes, worked sizing and cost calculations, and runnable configs across the Inference Engineering course, from GPUs and kernels to quantization, speculative decoding, disaggregation and production autoscaling.