01What a mixture of experts replaces

In a standard Transformer block, every token passes through the same single feed-forward sub-network. A mixture of experts (MoE) swaps that one block for several parallel "expert" sub-networks plus a small gating network — usually called the router — that decides which experts handle a given token. The original idea goes back to Jacobs, Jordan, Nowlan & Hinton (1991), where multiple expert networks each learned a subset of cases and a gating network combined them; Shazeer et al. (2017) introduced the modern sparsely-gated MoE layer that made this practical inside deep networks.

One caution worth internalising up front: an "expert" here is a sub-network inside one model, not a separately trained standalone model. There is no "law expert" or "medical expert" you can point to — the experts specialise in subtle, learned ways, and which token goes where is decided automatically by the router.

• MoE replaces one dense feed-forward block with many expert sub-networks + a router that picks which experts run.
• The concept dates to Jacobs et al. (1991); the modern sparsely-gated layer is from Shazeer et al. (2017).
• "Experts" are sub-networks within one model — not full, independent models.

02The key idea: sparse activation

Here is the single most important distinction in this whole lesson, and the one people most often get wrong: total parameters are not the same as active parameters. The router selects only a small number of experts — the top-k — for each token, so only a fraction of the model's parameters actually run for any given token. This is conditional computation: the network decides, per token, which parts of itself to use.

The payoff is that MoE decouples a model's total parameter count from its per-token compute cost. Total capacity can grow very large while the FLOPs spent on each token stay roughly constant. As a concrete reference point, the Mixtral paper (Jiang et al., 2024) describes a model with 8 experts per layer where the router selects 2 per token — so only part of the network runs for any one token even though all eight experts must be held in memory.

• Total parameters = the sum across all experts. Active parameters = only the experts the router picked for this token.
• Selecting top-k experts per token is sparse, conditional computation — the model uses different parts of itself for different tokens.
• The result: capacity scales up; per-token compute stays roughly flat (per Shazeer 2017; GLaM, Du et al. 2021).

03See the router decide

A row of tokens flows into the gating network on the left. For each token, the router scores all N experts and lights up only its top-k — those experts process the token; the rest stay dark. Change k and the routing scheme, then press Route the tokens and watch which experts wake up. The counters show how active parameters per token stay small even as total parameters (all experts) grow. The figures here are illustrative, not measured from any specific model — they exist to make the sparsity idea tangible.

04How routing works — and how it broke and got fixed

Routing is where most of the engineering lives, because a naive router has a nasty failure mode: it can collapse onto a few popular experts, leaving the rest barely trained and wasting the model's capacity. Different papers attack this differently, and it helps to state which scheme a claim refers to rather than treating MoE routing as one monolithic thing.

• GShard (Lepikhin et al., 2020) — used top-2 routing with automatic sharding to scale sparsely-gated MoE Transformers across many accelerators.
• Switch Transformer (Fedus et al., 2021) — simplified routing all the way to top-1 (one expert per token) to cut communication and complexity.
• Load-balancing loss — an auxiliary term added during training to push the router toward even expert utilisation, the standard remedy for collapse.
• BASE Layers (Lewis et al., 2021) — framed token-to-expert assignment as a balanced linear assignment problem, removing the auxiliary loss and its extra hyperparameters.
• Expert Choice routing (Zhou et al., 2022) — inverted the selection so experts pick their top-k tokens, guaranteeing balanced load and allowing a variable number of experts per token.
• DeepSeek-V3 (2024) — introduced an auxiliary-loss-free load-balancing strategy, balancing experts without the usual extra loss term.

Two more design ideas are worth knowing. DeepSeekMoE (Dai et al., 2024) improves expert specialisation through fine-grained expert segmentation plus shared (always-on) experts that capture common knowledge so the routed experts can specialise without redundancy. And because sparse models can be unstable to train and uncertain to fine-tune, ST-MoE (Zoph et al., 2022) contributed stabilising design choices (such as a router z-loss) to make sparse expert models stable and transferable.

05Why teams use MoE — and what it costs

The appeal is straightforward: for a given quality target, an MoE model can pretrain faster and infer faster than a dense model of the same total size, because each token only touches a slice of the parameters. GLaM (Du et al., 2021) reported a sparsely-activated MoE language model reaching comparable quality to dense models at far lower training and inference compute and energy. But sparsity is not free.

• Memory: all experts must be held in memory even though only a few run per token — so serving needs enough RAM/VRAM for the total parameters, not just the active ones (per Hugging Face's explainer).
• Routing & communication complexity: sending tokens to the right experts — often across devices — adds engineering and communication overhead.
• Training stability & fine-tuning: sparse models can be harder to train stably and to fine-tune well (ST-MoE, 2022, exists precisely to address this).
• The recurring trap: never quote total parameters as if they were the compute cost. State total vs active explicitly — conflating them is the most common MoE misconception.

Production systems lean on these tradeoffs deliberately. DeepSeek-V2 (2024) combined DeepSeekMoE with latent attention for an economical, efficient model, and DeepSeek-V3 (2024) scaled to a large MoE where most parameters stay inactive per token — the whole point of the architecture, turned into a serving strategy.

06Check your understanding

07Take it with you & go deeper