MoE vs dense: add experts, not compute per token

A dense transformer runs every parameter, the numbers it learns, for every token. A mixture-of-experts (MoE) model replaces each feed-forward block with several copies called experts, plus a small router that sends each token to only a few of them. That splits a model's total size from the compute each token costs.

1. 1
   The clip starts from a dense 7B model: 32 layers and 6.74 billion parameters, with 135 million in each layer's feed-forward block. At E = 1 there is one block and no router, so total and active parameters are both 6.74B. Each expert here is a full copy of that block, a what-if on the 7B shape rather than a released model.
2. 2
   The router is a linear layer from the token's 4,096 numbers to one score per expert, followed by a softmax, and it picks the top 2 experts for each token. At E = 2 both are always used, so 11.1B parameters are active. From there on the active count stays at 11.1B however many experts there are, because each token still visits 2. The router adds only 4,096 × E parameters per layer.
3. 3
   Total parameters grow with every expert: 37.0B at E = 8, 145B at E = 33 and 279B at E = 64, 25 times the active count. Compute per token, about 2 operations per active parameter, stays near 22 billion. Released models follow the same pattern: Mixtral 8x7B has 47B parameters with 13B active per token, and DeepSeek-V3 has 671B with 37B active.