Introduction

What would happen if instead of one big neural network you had hundreds of smaller ones — each specializing in different kinds of inputs — and a smart receptionist who decided which specialist to consult for each question? That is Mixture of Experts (MoE) in one sentence.

The idea is older than the deep-learning era. In 1991 Robert Jacobs and Michael Jordan proposed training separate "expert" networks alongside a "gating network" that learned to blend their outputs. The insight: divide a hard problem among specialists rather than forcing a single monolithic model to handle everything.

What changed everything is scale. In 2017 Noam Shazeer and colleagues at Google introduced the Sparsely-Gated Mixture-of-Experts layer that only activates a small subset — say top-2 out of 64 — of expert subnetworks for each input token. The rest stay idle. Suddenly you could have a model with 137 billion parameters but pay the compute cost of a much smaller one.

That trade-off — large capacity, small active cost — is why MoE architectures power some of the biggest models ever deployed, including GPT-4 (reportedly a 16-expert MoE) and Google's Gemini Ultra. Understanding MoE means understanding how AI escaped the compute wall that would otherwise have stopped scaling cold.

Route the Token

Below are four experts and a gating network. Type any word or pick a preset, then press Route — the gate assigns a score to each expert and the top-2 are activated. Each expert has learned a simple specialty (sentiment, length, numbers, punctuation). Watch how different inputs consistently land on different expert pairs.

<p class="hint">A gating network scores each expert and activates the <b>top-2</b>.
    Type a word or pick a preset, then press <b>Route</b>.</p>
<div class="presets">
  <button class="preset" data-v="hello">hello</button>
  <button class="preset" data-v="12345">12345</button>
  <button class="preset" data-v="URGENT!!!">URGENT!!!</button>
  <button class="preset" data-v="sad">sad</button>
  <button class="preset" data-v="cat">cat</button>
</div>
<div class="input-row">
  <input id="token" type="text" value="hello" maxlength="40" placeholder="type a word…" />
  <button id="route-btn" type="button">Route</button>
</div>
<div id="experts" class="experts"></div>
<div id="output" class="output"></div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
.presets { display: flex; flex-wrap: wrap; gap: .35rem; margin-bottom: .6rem; }
.preset { font: 500 12px system-ui; padding: .25rem .55rem; border: 1px solid #adb1b8;
          background: #f0f2f4; color: #1d3557; border-radius: 6px; cursor: pointer; }
.preset:hover { background: #dde2e8; }
.input-row { display: flex; gap: .5rem; margin-bottom: .8rem; }
#token { flex: 1; font: 15px system-ui; padding: .4rem .6rem; border: 1px solid #adb1b8;
         border-radius: 8px; }
#route-btn { font: 600 14px system-ui; padding: .4rem .9rem; background: #1d3557;
             color: #fff; border: none; border-radius: 8px; cursor: pointer; }
#route-btn:hover { background: #274472; }
.experts { display: grid; grid-template-columns: repeat(2, 1fr); gap: .6rem; margin-bottom: .7rem; }
.expert { border: 2px solid #cdd9e3; border-radius: 10px; padding: .55rem .7rem;
          transition: all .25s; background: #f7f9fb; }
.expert.active { border-color: #1d7a3a; background: #eafaf0; }
.expert.inactive { opacity: .45; }
.expert-name { font-weight: 700; font-size: .93rem; margin-bottom: .25rem; }
.expert-spec { font-size: .78rem; color: #555; margin-bottom: .35rem; }
.bar-wrap { height: 10px; background: #dde2e8; border-radius: 5px; overflow: hidden; }
.bar { height: 100%; border-radius: 5px; background: #4a90d9; transition: width .3s; }
.bar.top { background: #1d7a3a; }
.score-label { font-size: .75rem; color: #444; margin-top: .2rem; }
.active-badge { display: inline-block; font-size: .7rem; font-weight: 700;
                background: #1d7a3a; color: #fff; border-radius: 4px; padding: 1px 5px;
                margin-left: .35rem; vertical-align: middle; }
.output { font-size: .88rem; background: #eef2f7; border-radius: 8px; padding: .5rem .7rem;
          min-height: 2rem; line-height: 1.5; }
.output b { color: #1d3557; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Four lightweight expert feature extractors + a linear gating network.
// Each expert computes a scalar score from simple string features; the gate turns the four raw scores into probabilities (softmax) and picks the top-2.

const EXPERTS = [
  {
    name: 'Expert 1 – Sentiment',
    spec: 'Handles emotionally charged words',
    // Features: presence of common positive/negative words, exclamation marks
    score(tok) {
      const t = tok.toLowerCase();
      const pos = ['good','great','love','happy','joy','yes','hello','hi'].filter(w=>t.includes(w)).length;
      const neg = ['bad','sad','hate','no','ugh','cry','fail','angry','wrong'].filter(w=>t.includes(w)).length;
      const excl = (tok.match(/!/g)||[]).length;
      return pos * 1.8 + neg * 1.8 + excl * 1.2;
    }
  },
  {
    name: 'Expert 2 – Numeric',
    spec: 'Handles tokens containing digits or math',
    score(tok) {
      const digits = (tok.match(/\d/g)||[]).length;
      const ops = (tok.match(/[+\-*/=<>]/g)||[]).length;
      return digits * 2.0 + ops * 1.5;
    }
  },
  {
    name: 'Expert 3 – Length',
    spec: 'Handles very short or very long tokens',
    score(tok) {
      const l = tok.length;
      // Short (<= 3) or long (>= 10) get high scores
      if (l <= 3) return 4 - l;
      if (l >= 10) return l - 8;
      return 0;
    }
  },
  {
    name: 'Expert 4 – Punctuation',
    spec: 'Handles tokens heavy in symbols',
    score(tok) {
      const syms = (tok.match(/[^a-zA-Z0-9\s]/g)||[]).length;
      const caps = (tok.match(/[A-Z]/g)||[]).length;
      return syms * 2.2 + caps * 0.6;
    }
  }
];

function softmax(arr) {
  const max = Math.max(...arr);
  const exps = arr.map(x => Math.exp(x - max));
  const sum = exps.reduce((a,b) => a+b, 0);
  return exps.map(x => x / sum);
}

function route(token) {
  const raw = EXPERTS.map(e => e.score(token));
  const probs = softmax(raw);
  // top-2 indices
  const ranked = probs.map((p,i) => [p,i]).sort((a,b) => b[0]-a[0]);
  const top2 = new Set([ranked[0][1], ranked[1][1]]);
  return { probs, top2 };
}

// Node.js verification harness (runs only when 'document' is absent)
if (typeof document === 'undefined') {
  // Verify top-2 routing for known inputs
  const cases = [
    { tok: '12345', expectTop: 1 },   // digits -> Expert 2 (Numeric) should be #1
    { tok: 'URGENT!!!', expectTop: 3 }, // symbols+caps -> Expert 4 (Punctuation) should be #1
    { tok: 'sad', expectTop: 0 },      // sentiment word -> Expert 1 should be #1
    { tok: 'hi', expectTop: 2 },       // short -> Expert 3 (Length) should be #1
  ];
  let pass = true;
  cases.forEach(({ tok, expectTop }) => {
    const raw = EXPERTS.map(e => e.score(tok));
    const probs = softmax(raw);
    const ranked = probs.map((p,i) => [p,i]).sort((a,b) => b[0]-a[0]);
    const top1 = ranked[0][1];
    const ok = top1 === expectTop;
    console.log(fmt('%v% "%tok%" top expert: %name%%v2%', { v: ok ? '✓' : '✗', tok: tok, name: EXPERTS[top1].name, v2: ok ? '' : ' (expected ' + EXPERTS[expectTop].name + ')' }));
    if (!ok) pass = false;
  });
  // Verify softmax sums to 1
  const probs = softmax([1,2,3,4]);
  const sum = probs.reduce((a,b)=>a+b,0);
  const sumOk = Math.abs(sum - 1) < 1e-10;
  console.log(fmt('%v% softmax sums to 1: %sum%', { v: sumOk ? '✓' : '✗', sum: sum }));
  if (!sumOk) pass = false;
  console.log(pass ? 'ALL TESTS PASSED' : 'SOME TESTS FAILED');
  process.exit(pass ? 0 : 1);
} else {
  // Browser: render UI
  function render(token, result) {
    const { probs, top2 } = result;
    const container = document.getElementById('experts');
    container.innerHTML = '';
    EXPERTS.forEach((exp, i) => {
      const isActive = top2.has(i);
      const pct = (probs[i] * 100).toFixed(1);
      const div = document.createElement('div');
      div.className = 'expert ' + (isActive ? 'active' : 'inactive');
      div.innerHTML =
        fmt('<div class="expert-name">%name%%v%</div><div class="expert-spec">%spec%</div><div class="bar-wrap"><div class="bar%v2%" style="width:%pct%%"></div></div><div class="score-label">Gate weight: %pct%%</div>', { name: exp.name, v: isActive ? '<span class="active-badge">ACTIVE</span>' : '', spec: exp.spec, v2: isActive ? ' top' : '', pct: pct });
      container.appendChild(div);
    });
    const activeNames = [...top2].map(i => EXPERTS[i].name.split('–')[1].trim());
    document.getElementById('output').innerHTML =
      fmt('Token <b>"%token%"</b> routed to <b>%activeNames%</b>. %v% expert%v2% inactive — zero compute wasted.', { token: token, activeNames: activeNames.join('</b> + <b>'), v: EXPERTS.length - 2, v2: EXPERTS.length - 2 > 1 ? 's' : '' });
  }

  document.getElementById('route-btn').addEventListener('click', function() {
    var tok = document.getElementById('token').value.trim() || 'hello';
    render(tok, route(tok));
  });

  document.querySelectorAll('.preset').forEach(function(btn) {
    btn.addEventListener('click', function() {
      var v = btn.dataset.v;
      document.getElementById('token').value = v;
      render(v, route(v));
    });
  });

  render('hello', route('hello'));
}
Result
Console

Notice that only 2 of 4 experts fire for any token. In a real MoE transformer layer there might be 64 or even 2048 experts, but still only 2 or 4 activate per token. The inactive experts contribute nothing to the forward pass — their weights are not touched, no multiply-add operations are wasted. That is the whole trick: capacity without compute.

The Real Complexity

The MoE trick sounds almost too good to be true. Here is what the mathematics actually guarantees — and what it does not.

The compute win is real. If a dense model has N parameters and processes every token through all of them, an MoE model with kN parameters but top-r routing uses roughly r/k of the compute of the equivalent dense model. GPT-4 reportedly has ~1.8 trillion parameters across 16 experts but activates roughly 220 billion per token — about 12 % of the total.

But routing is hard to train. The gate produces a probability distribution over experts. The naive "pick the top-k" operation is not differentiable — the experts that lose do not receive gradients. Shazeer et al. fixed this with a noisy top-k gating scheme: add Gaussian noise before the softmax so that near-tied experts swap rank randomly during training, giving every expert a chance to learn.

Load balancing is the villain. If one expert is slightly better early in training, the gate routes more tokens to it. It gets more gradient, becomes better still, and — without intervention — expert collapse occurs: nearly all tokens go to one expert and the rest go dormant. Real MoE systems add an auxiliary loss that penalizes unequal loads, forcing the gate to spread traffic.

Memory stays expensive. All expert weights must live somewhere. MoE models are "cheap to run but expensive to store" — their multi-terabyte footprints require careful model parallelism across many accelerators. This is why you cannot simply download GPT-4 and run it on your laptop.

For a deeper look at how neural networks learn in the first place, see the article on neural network training. To understand how scaling laws govern what size of model you need, the broader context of P vs NP shows why efficient computation always matters.

Where It Matters

Mixture of Experts is not an academic curiosity — it is the dominant paradigm for the largest AI models in production today:

  • Large language models: GPT-4, Gemini Ultra, Mistral's Mixtral 8×7B and 8×22B, and many others are MoE models. The 2024 open-source Mixtral 8×7B matches GPT-3.5 quality while activating only 12.9 billion parameters per token from a 46.7-billion total.
  • Vision and multimodal AI: MoE layers are being adopted in vision transformers (ViT-MoE) and multimodal models to handle the diversity of image patches without enlarging the dense per-token compute budget.
  • Scientific computing: protein-structure models and climate simulators use MoE-style sparse attention to handle heterogeneous input types (different residue chemistries, different atmospheric variables) through specialized subnetworks.
  • Recommendation systems: industry-scale recommender systems at Google, Meta, and Alibaba have used MoE for years — long before the LLM wave — to handle the enormous diversity of user-item interaction patterns with tractable compute.
  • Efficient inference: because inactive experts can be offloaded to CPU or disk, MoE enables running very large models on limited hardware — a key advantage for on-device or edge AI.

The pattern is always the same: heterogeneous inputs, limited compute, desire for large capacity. MoE solves the triangle by specializing and routing rather than by making every neuron process everything.

Conclusion

Mixture of Experts answers one of the hardest questions in modern AI: how do you make a model smarter without making it proportionally more expensive to run? The answer — divide labor among specialists and route each input to the right few — is elegant and, once you see it, almost obvious.

The deep insight is that intelligence may not require every neuron to participate in every thought. Just as a hospital runs efficiently because cardiologists handle hearts and neurologists handle brains, a model runs efficiently when routing keeps each expert focused on the inputs it handles best.

The remaining challenges — load balancing, memory footprint, routing stability — are active research areas. But the core idea is settled: sparse, expert-routed computation is not a workaround. It is an architecture. And it is why the models answering your questions today can be a hundred times larger than anything that ran five years ago, at a compute cost that is merely a few times higher.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/mixture-of-experts/Content licensed under CC BY-NC 4.0.