Lensa ML
Lensa ML

Mixture of Experts

Step 1 of 6: Bigger Model, Same Compute

Many experts, few active per token

Number of experts E (top-k = 2)8
One layer: the router sends each token to 2 of 8 expertstoken hrouter: 4096 → 8, softmaxE1E2E3E4E5E6E7E8Whole model, 32 layers (bar scale: E = 64)Total37.0BActive11.1BWhat-if on the 7B shape: each expert is a full FFN, 3 × 4096 × 11008 = 135M
Total params
37.0B
Active per token
11.1B
Total ÷ active
3.3×

With 8 experts, the model holds 37.0B parameters, but a token only runs 2 experts per layer, so it uses 11.1B. Top-2 of full-size experts costs about twice the dense FFN, whatever E is.

Drag the number of experts E from 1 to 64 and compare total parameters with active ones. Only the feed-forward block is copied. Attention, embeddings and norms stay shared, so the total grows by about 135M parameters per layer for each new expert, not by a whole 7B model. The idea goes back to Shazeer et al. (2017), who placed an MoE layer with up to thousands of experts between LSTM layers and reached 137B parameters. Newer models often use many small experts instead of a few full-size ones.

Step 2 of 6: The Router

A linear layer and a softmax over experts

softmax(W_r · h) for " France" over 8 experts0.00.51.0uniform 1/80.01E10.01E20.01E30.01E40.02E50.02E60.01E70.90E8W_r is 4096 × 8 = 32,768 weights per layer · logits illustrative
Top expert
E8
Its probability
0.90
Router entropy
0.72 bits

The router is confident about " France": E8 gets 90% and the entropy is 0.72 of a possible 3 bits.

Click each token of the prompt and compare how sharply the router picks an expert. The router computes logits = h · W_r and a softmax, so its cost is tiny next to the experts it chooses. Each of the 32 layers has its own router, so one token can visit different experts at every depth. Shazeer et al. added tuned Gaussian noise to these logits during training to spread the load. In Mixtral, experts did not split by topic: routing followed syntax more than subject, and consecutive tokens often went to the same expert.

Step 3 of 6: Top-k Routing

Pick k experts, mix their outputs

Experts per token k2
Router picks the top 2; gates = softmax over the chosen logitsE1p=0.04E2p=0.08E3g=0.61E4g=0.39E5p=0.05E6p=0.05E7p=0.16E8p=0.060.61 × E3(h)0.110.46-0.48-0.690.39 × E4(h)0.530.04-0.61-0.26y = Σ g · E(h)0.270.30-0.53-0.52Toy 4-dimensional expert outputs (real ones have 4096 dims) · logits illustrative
Active experts
2 of 8
Gate weights
0.61 / 0.39
Active FFN share
25.0%

The 2 chosen experts held only 56% of the router's probability. After renormalizing, their gates sum to 1 and y blends their outputs in those proportions.

Drag Experts per token k from 1 to 4, then switch tokens to see the gates change. Mixtral writes the layer as y = Σ softmax(TopK(h · W_g))_i · E_i(h), which is the renormalized mix shown here. Shazeer et al. assumed k must be at least 2 for the router to learn useful gradients. Switch Transformer showed top-1 works if the output is scaled by the router probability p_i instead of a renormalized 1.0, since the gradient reaches the router through p_i. Compute per token grows linearly with k.

Step 4 of 6: Load Balancing

Keeping every expert busy

Auxiliary loss weight α0.000
256 tokens × top-2 = 512 routing slots per expert0100200balanced: 64229E1165E279E327E411E51E6idleE7idleE8Simplified model: α shrinks the router's built-in bias to ×1.00 · seeded logits
Busiest expert's share
44.7%
Idle experts
2
Aux loss E·Σ f·P
2.49

With no balancing loss the router favors E1 and E2, which take 77% of the slots, and 2 experts sit idle. The aux loss is 2.49, far above the balanced value of about 1.

Drag the auxiliary loss weight α from 0 toward 0.1 and watch the histogram flatten. Without help, routing tends to collapse: experts that win early get more training and win more. Switch Transformer adds α · N · Σ f_i · P_i to the loss, where f_i is the fraction of tokens sent to expert i and P_i is its mean router probability. Only P_i is differentiable. The term equals 1 under uniform routing. The paper used α = 0.01. DeepSeek-V3 drops this loss and nudges a per-expert routing bias up or down instead.

Step 5 of 6: Capacity & Dropped Tokens

What happens when an expert is full

Capacity factor1.00
Capacity C = 1.00 × 256 × 2 / 8 = 64 slots per expert050100−40E1−40E2−14E355E450E546E650E725E8C = 64Routing from step 4 at α = 0.05 · red = over capacity, skipped by the expert
Capacity per expert
64
Slots dropped
94
Percent dropped
18.4%
Tokens with no expert
14

At factor 1.00, only the busiest experts overflow. 94 slots are dropped (18.4%); 14 tokens skip the layer's FFN entirely and keep the residual value.

Drag the capacity factor between 0.5 and 2.0 and watch the red overflow shrink. Hardware favors fixed tensor shapes, so each expert gets a buffer of C = factor × tokens × k / E slots per batch. A factor above 1 leaves room for uneven routing but wastes compute on empty slots. Switch Transformer reported dropping under 1% of tokens with its balancing loss in place. MegaBlocks (Gale et al., 2022) avoids drops with block-sparse kernels that handle any number of tokens per expert, and DeepSeek-V3 drops none in training or inference.

Step 6 of 6: MoE in Practice

Switch, Mixtral and DeepSeek-V3

One MoE layer: 8 routed experts, 2 routed active per tokenE1E2E3E4E5E6E7E8Weights at fp16 (2 bytes per parameter)Held in memory93 GBRead per token26 GBRouted experts: 32 layers × 8 × 176M = 45.1B (97% of all weights)Computed from the Mixtral paper's Table 1 configuration · dense 7B: 13 GB
Total params
46.7B
Active per token
12.9B
Active share
27.6%
fp16 weights
93 GB

Each token reads 12.9B of 46.7B weights, not 8 × 7B = 56B, because attention and embeddings are shared across experts. Serving it still takes 93 GB at fp16, more than one 80 GB GPU.

Click Switch-style, Mixtral 8x7B and DeepSeek-V3 and compare the two memory bars. The name 8x7B suggests 56B, but Mixtral has about 46.7B parameters because only the feed-forward blocks are copied. Switch-C, the largest Switch model, reached 1.6T parameters with 2,048 experts. Active parameters set the FLOPs per token, not the memory bill. In a large batch, different tokens pick different experts, so most experts get read on every step. Serving systems therefore place experts on separate GPUs and send tokens between them, a scheme called expert parallelism.

What is a mixture of experts (MoE) model?

A mixture of experts (MoE) model replaces one large feed-forward layer with many smaller networks, called experts, and a router that sends each token to only a few of them. The model can hold far more parameters than it uses for any single token. Compute per token depends on how many experts run, while total capacity depends on how many exist.

MoE layers sit inside the transformer blocks of many large language models, including Switch Transformer, Mixtral 8x7B and DeepSeek-V3. They let a lab train a model with hundreds of billions of parameters at roughly the per-token cost of a much smaller dense model. The price is memory and engineering: every expert must be stored, the router has to spread work evenly, and the experts are usually split across many GPUs.

Key ideas

  • An MoE layer replaces a transformer's feed-forward block with many experts and a router.
  • The router is a linear layer and a softmax that scores every expert for each token.
  • Only the top-k experts run, and their outputs are mixed by the gate weights.
  • Total parameters grow with the number of experts, while compute per token grows with k.
  • An auxiliary loss or a routing bias keeps the load spread evenly across experts.
  • Every expert must sit in memory, so MoE models trade memory for cheaper compute per token.

Common questions

Is an MoE model the same as an ensemble?

No. An ensemble runs several full models on every input and averages their predictions. An MoE runs one model in which each layer picks a few experts per token, and the experts are trained together with the router. The choice happens inside the network, token by token, not once per input.

Do MoE experts specialize in topics like math or code?

Not in a way people would recognize. The Mixtral authors found that expert choice was almost the same for arXiv papers, biology abstracts and philosophy texts. Routing followed token type and syntax more than subject, so an expert is better seen as a learned slice of the feed-forward layer than as a specialist.

Why are MoE models hard to fine-tune and serve?

The full parameter set must fit in GPU memory even though few experts run per token. Routing can shift during fine-tuning, which can unbalance the load or overfit small datasets. Serving needs expert parallelism and communication between GPUs, and small batches leave most experts idle while still holding them in memory.

Does MoE make inference faster?

Per token, an MoE does about as much arithmetic as a dense model of its active size, so it can be faster than a dense model of its total size. At small batch sizes decoding is limited by memory reads, and reading a few experts is cheaper than reading a full dense layer. At large batch sizes most experts are read anyway, and the speed gain depends on how well the system spreads tokens across GPUs.

Related explainers

References

  1. Shazeer et al. (2017), "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer"
  2. Fedus, Zoph and Shazeer (2021), "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity"
  3. Jiang et al. (2024), "Mixtral of Experts"
  4. DeepSeek-AI (2024), "DeepSeek-V3 Technical Report"
  5. Gale et al. (2022), "MegaBlocks: Efficient Sparse Training with Mixture-of-Experts"

Last updated