Speculative decoding

Decode is slow for a reason that has little to do with arithmetic. At small batch sizes, each decode step reads every weight of the model out of GPU memory to produce a single token, and the compute cores spend most of that step waiting for the bytes to arrive. A forward pass over five tokens reads the same weights once and does five tokens' worth of math, so it costs barely more wall-clock time than a pass over one. The spare compute is sitting there, unused.

Speculative decoding spends it. A cheap proposer guesses the next few tokens. The model you actually serve, the target, runs one forward pass over all of those guesses at once and checks each one. Every guess that matches what the target would have produced is kept, and the target contributes one more token of its own at the first disagreement. When the guesses are good, one expensive step emits three or four tokens instead of one.