Number formats for inference

A model's weights are just numbers, and how many bits you spend on each one decides three things at once: how much GPU memory the model needs, how many bytes decode must read per token, and how faithfully each weight is represented. Models are trained in 16-bit formats. Serving them in 8 or 4 bits halves or quarters the first two, and the art is keeping the third close enough that nobody can tell.

Two ideas make that possible. A floating-point format splits its bits between an exponent, which sets range, and a mantissa, which sets precision. And a small format need not cover every possible value on its own: a shared scale factor, stored once per tensor, row or block, stretches it to fit the values actually present.