Mixture of Experts

What would happen if instead of one big neural network you had hundreds of smaller ones — each specializing in different kinds of inputs — and a smart receptionist who decided which specialist to consult for each question? That is Mixture of Experts (MoE) in one sentence.

The idea is older than the deep-learning era. In 1991 Robert Jacobs and Michael Jordan proposed training separate "expert" networks alongside a "gating network" that learned to blend their outputs. The insight: divide a hard problem among specialists rather than forcing a single monolithic model to handle everything.

What changed everything is scale. In 2017 Noam Shazeer and colleagues at Google introduced the Sparsely-Gated Mixture-of-Experts layer that only activates a small subset — say top-2 out of 64 — of expert subnetworks for each input token. The rest stay idle. Suddenly you could have a model with 137 billion parameters but pay the compute cost of a much smaller one.

That trade-off — large capacity, small active cost — is why MoE architectures power some of the biggest models ever deployed, including GPT-4 (reportedly a 16-expert MoE) and Google's Gemini Ultra. Understanding MoE means understanding how AI escaped the compute wall that would otherwise have stopped scaling cold.

Below are four experts and a gating network. Type any word or pick a preset, then press Route — the gate assigns a score to each expert and the top-2 are activated. Each expert has learned a simple specialty (sentiment, length, numbers, punctuation). Watch how different inputs consistently land on different expert pairs.

<p class="hint">{{hint}}</p>
<div class="presets">
  <button class="preset" data-v="hello">hello</button>
  <button class="preset" data-v="12345">12345</button>
  <button class="preset" data-v="URGENT!!!">URGENT!!!</button>
  <button class="preset" data-v="sad">sad</button>
  <button class="preset" data-v="cat">cat</button>
</div>
<div class="input-row">
  <input id="token" type="text" value="hello" maxlength="40" placeholder="{{placeholder}}" />
  <button id="route-btn" type="button">{{route_btn}}</button>
</div>
<div id="experts" class="experts"></div>
<div id="output" class="output"></div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
.presets { display: flex; flex-wrap: wrap; gap: .35rem; margin-bottom: .6rem; }
.preset { font: 500 12px system-ui; padding: .25rem .55rem; border: 1px solid #adb1b8;
          background: #f0f2f4; color: #1d3557; border-radius: 6px; cursor: pointer; }
.preset:hover { background: #dde2e8; }
.input-row { display: flex; gap: .5rem; margin-bottom: .8rem; }
#token { flex: 1; font: 15px system-ui; padding: .4rem .6rem; border: 1px solid #adb1b8;
         border-radius: 8px; }
#route-btn { font: 600 14px system-ui; padding: .4rem .9rem; background: #1d3557;
             color: #fff; border: none; border-radius: 8px; cursor: pointer; }
#route-btn:hover { background: #274472; }
.experts { display: grid; grid-template-columns: repeat(2, 1fr); gap: .6rem; margin-bottom: .7rem; }
.expert { border: 2px solid #cdd9e3; border-radius: 10px; padding: .55rem .7rem;
          transition: all .25s; background: #f7f9fb; }
.expert.active { border-color: #1d7a3a; background: #eafaf0; }
.expert.inactive { opacity: .45; }
.expert-name { font-weight: 700; font-size: .93rem; margin-bottom: .25rem; }
.expert-spec { font-size: .78rem; color: #555; margin-bottom: .35rem; }
.bar-wrap { height: 10px; background: #dde2e8; border-radius: 5px; overflow: hidden; }
.bar { height: 100%; border-radius: 5px; background: #4a90d9; transition: width .3s; }
.bar.top { background: #1d7a3a; }
.score-label { font-size: .75rem; color: #444; margin-top: .2rem; }
.active-badge { display: inline-block; font-size: .7rem; font-weight: 700;
                background: #1d7a3a; color: #fff; border-radius: 4px; padding: 1px 5px;
                margin-left: .35rem; vertical-align: middle; }
.output { font-size: .88rem; background: #eef2f7; border-radius: 8px; padding: .5rem .7rem;
          min-height: 2rem; line-height: 1.5; }
.output b { color: #1d3557; }

// Code not found

Notice that only 2 of 4 experts fire for any token. In a real MoE transformer layer there might be 64 or even 2048 experts, but still only 2 or 4 activate per token. The inactive experts contribute nothing to the forward pass — their weights are not touched, no multiply-add operations are wasted. That is the whole trick: capacity without compute.

The MoE trick sounds almost too good to be true. Here is what the mathematics actually guarantees — and what it does not.

The compute win is real. If a dense model has N parameters and processes every token through all of them, an MoE model with kN parameters but top-r routing uses roughly r/k of the compute of the equivalent dense model. GPT-4 reportedly has ~1.8 trillion parameters across 16 experts but activates roughly 220 billion per token — about 12 % of the total.

But routing is hard to train. The gate produces a probability distribution over experts. The naive "pick the top-k" operation is not differentiable — the experts that lose do not receive gradients. Shazeer et al. fixed this with a noisy top-k gating scheme: add Gaussian noise before the softmax so that near-tied experts swap rank randomly during training, giving every expert a chance to learn.

Load balancing is the villain. If one expert is slightly better early in training, the gate routes more tokens to it. It gets more gradient, becomes better still, and — without intervention — expert collapse occurs: nearly all tokens go to one expert and the rest go dormant. Real MoE systems add an auxiliary loss that penalizes unequal loads, forcing the gate to spread traffic.

Memory stays expensive. All expert weights must live somewhere. MoE models are "cheap to run but expensive to store" — their multi-terabyte footprints require careful model parallelism across many accelerators. This is why you cannot simply download GPT-4 and run it on your laptop.

For a deeper look at how neural networks learn in the first place, see the article on neural network training. To understand how scaling laws govern what size of model you need, the broader context of P vs NP shows why efficient computation always matters.

Mixture of Experts is not an academic curiosity — it is the dominant paradigm for the largest AI models in production today:

• Large language models: GPT-4, Gemini Ultra, Mistral's Mixtral 8×7B and 8×22B, and many others are MoE models. The 2024 open-source Mixtral 8×7B matches GPT-3.5 quality while activating only 12.9 billion parameters per token from a 46.7-billion total.
• Vision and multimodal AI: MoE layers are being adopted in vision transformers (ViT-MoE) and multimodal models to handle the diversity of image patches without enlarging the dense per-token compute budget.
• Scientific computing: protein-structure models and climate simulators use MoE-style sparse attention to handle heterogeneous input types (different residue chemistries, different atmospheric variables) through specialized subnetworks.
• Recommendation systems: industry-scale recommender systems at Google, Meta, and Alibaba have used MoE for years — long before the LLM wave — to handle the enormous diversity of user-item interaction patterns with tractable compute.
• Efficient inference: because inactive experts can be offloaded to CPU or disk, MoE enables running very large models on limited hardware — a key advantage for on-device or edge AI.

The pattern is always the same: heterogeneous inputs, limited compute, desire for large capacity. MoE solves the triangle by specializing and routing rather than by making every neuron process everything.

Mixture of Experts answers one of the hardest questions in modern AI: how do you make a model smarter without making it proportionally more expensive to run? The answer — divide labor among specialists and route each input to the right few — is elegant and, once you see it, almost obvious.

The deep insight is that intelligence may not require every neuron to participate in every thought. Just as a hospital runs efficiently because cardiologists handle hearts and neurologists handle brains, a model runs efficiently when routing keeps each expert focused on the inputs it handles best.

The remaining challenges — load balancing, memory footprint, routing stability — are active research areas. But the core idea is settled: sparse, expert-routed computation is not a workaround. It is an architecture. And it is why the models answering your questions today can be a hundred times larger than anything that ran five years ago, at a compute cost that is merely a few times higher.