Quantizing without losing quality

Quantization is the cheapest performance win in inference and the easiest one to get subtly wrong. Converting Kite's tab-completion model from BF16 to FP8 halves its weight memory and the bytes each decode step reads, and on most models nobody will notice any change in the output. On some models, some prompts, or at 4 bits, the answers get quietly worse, and the only defense is a disciplined process: choose what to quantize by sensitivity, calibrate with representative data, and measure quality with enough samples to believe the result.

Not every number in a model is equally fragile. From least to most sensitive: the weights of linear layers, then the activations flowing between them, then the KV cache, then the attention math itself, especially softmax. That ordering is the whole strategy. Start at the robust end and move toward the fragile end only when an evaluation says you can.