Picking the smallest model that passes

The cheapest inference optimization happens before you touch a GPU: serve a smaller model. Most techniques later in this course, from quantization to speculative decoding, win a factor of a few at best. Moving from an 8B model to a 1.5B model changes weight memory, compute per token, and the ceiling on decode speed by more than five times, all at once.

The catch is quality, and the answer is a test. Decide what "good enough" means as a number on an evaluation set built from your own traffic, then try candidates from smallest to largest and ship the first one that passes. The rest of this lesson is the arithmetic that makes the prize concrete and the details that make the test trustworthy.