Mixture of experts models

A mixture of experts (MoE) model replaces the single MLP in each transformer block with many smaller MLPs, called experts, plus a tiny router that picks a few of them for each token. Attention is unchanged and shared by every token. The result is a model with a large total parameter count and a much smaller active parameter count: the weights one token actually passes through.

Kite's chat model, Qwen/Qwen3-30B-A3B-Instruct-2507, is the running example. It has 30.5 billion parameters in total and 3.3 billion active per token, with 128 experts in each of its 48 layers and 8 chosen per token. The name says it: 30B total, A3B for about 3 billion active. For serving, the split matters because the two numbers drive different costs. Memory follows the total. Compute per token follows the active count.