Shipping changes, cost, and observability

An inference fleet is never finished. In a normal month Kite's team swaps in a newer checkpoint, tries a 4-bit build of the completion model, upgrades vLLM, changes --max-num-seqs, and edits the system prompt. Each of those can shift time to first token, inter-token latency, cost per request and answer quality, sometimes all four, and often in ways a unit test cannot see.

So operating the fleet comes down to three habits. Ship changes gradually, with gates that measure the things users feel. Know what a day of serving costs and what the alternative would cost. And keep a short list of metrics that tell you something is wrong before a customer does.