Autoscaling, routing, and queues

Autoscaling habits from web services break on GPUs for two reasons. A web pod starts in seconds, while a GPU replica takes minutes to go from "requested" to "serving", as the previous lesson showed. And a web pod degrades gradually as load rises, while an inference replica has a hard ceiling: the number of sequences it can batch before inter-token latency blows its budget or the KV cache fills. Scaling an inference fleet means knowing that ceiling, noticing early that you are approaching it, and having somewhere safe for requests to wait while new capacity boots.

Three pieces do this work: the autoscaler decides how many replicas exist, the router picks a replica for each request, and the queue holds what no replica should take right now.