Tensor, expert, and pipeline parallelism

A model is split across GPUs when its weights plus useful KV cache do not fit on one, or one GPU is too slow. There are three basic cuts. Tensor parallelism slices every weight matrix so each GPU holds a piece of every layer. Pipeline parallelism gives each GPU a consecutive run of whole layers. Expert parallelism, for mixture-of-experts models, places whole experts on different GPUs and sends each token to wherever its chosen experts live.

Each cut decides how often GPUs talk and how much they send, which must match the wire between them. Inside a node, NVLink gives each H100 900 GB/s of total bandwidth. Between nodes, a 400 Gb/s InfiniBand NIC carries 50 GB/s. The usual rule, tensor parallelism inside a node and pipeline or expert parallelism across nodes, follows from that arithmetic.