Nodes, interconnects, and MIG

A model that does not fit on one GPU has to be split, and every split creates communication. Where that communication travels decides whether the split is nearly free or ruinously slow. Inside a standard server node, eight GPUs talk over NVLink through NVSwitch chips at hundreds of gigabytes per second. Between nodes, traffic leaves through network cards at 400 to 800 gigabits per second. The gap between those two numbers shapes almost every multi-GPU deployment.

MIG (Multi-Instance GPU) is the same question in reverse. When a model is far smaller than the card, you can cut one GPU into as many as seven isolated instances rather than let most of it sit unused.