CUDA kernels, selection, and fusion

When Python code multiplies two tensors on a GPU, the GPU does not run Python. The CPU launches a kernel: a function compiled for the GPU that runs across thousands of threads at once, each handling one element or one block of the tensor. A forward pass through an LLM is a long sequence of these launches, one or more for every operation in the model.

Two decisions around each kernel set the speed. Selection is picking the best implementation of an operation for this GPU architecture, number format, and tensor shape. Fusion is replacing several small kernels, each of which reads its input from memory and writes its output back, with one kernel that does all the steps in a single pass. Since decode is memory-bound, and many small operations are too, fusion often matters more than raw arithmetic speed.