Matmul and activations for inference engineers

Strip away the names and a language model is a list of big matrices and a fixed recipe for multiplying a vector through them. The matrices are the weights. The vectors that flow between them, one per token, are the activations. Almost every number an inference engineer cares about falls out of those two facts: how much memory the model needs, how many operations each token costs, and why reading weights, not doing arithmetic, is often the slow part.

You do not need linear algebra beyond one rule. Multiplying a vector of length d_in by a matrix of shape d_out × d_in produces a vector of length d_out, and it takes one multiply and one add for every entry in the matrix. Hold on to that and a whole model becomes countable.