Containers and cold starts

A web service container starts in seconds because it is small and has almost nothing to load. An inference container is the opposite. It needs a GPU node, an image of 10 GB or more full of CUDA libraries, tens of gigabytes of weights, and an engine that captures and compiles kernels for the GPU it lands on. The time from "we need another replica" to "that replica answers a request" is the cold start. It decides how fast you can scale, how much idle capacity you have to pay for, and what happens to users during a traffic spike.

A cold start has five parts: get a GPU node, pull the image, load the weights, start and compile the engine, and warm up. They have different causes and different fixes, so the first job is to measure them separately.