Introduction

Imagine you measure the heights of a crowd, but nobody recorded who is a child and who is an adult. You strongly suspect there are two groups mixed together, each roughly bell-shaped. You want to find the average height of each group — but to do that you'd need to know who belongs to which group, and that is exactly the information you don't have.

This is a chicken-and-egg problem. If you knew the group labels, fitting each bell curve would be easy. If you knew the curves, guessing the labels would be easy. You have neither.

Expectation-Maximization (EM) breaks the deadlock with a wonderfully simple idea: make up an answer, then let it improve itself. Start with a rough guess for the two curves. Use it to assign each point a soft guess of which group it came from (the E-step). Then pretend those guesses are true and re-fit the curves (the M-step). Repeat. Each round fits the data at least as well as the last, and the picture sharpens until it locks into place.

EM was given its general form and name by Dempster, Laird and Rubin in 1977, and it is one of the workhorses behind clustering, topic models, and missing-data statistics today.

Watch It Converge

Below are points scattered along a line — they secretly came from two overlapping clusters, but the labels are hidden. Two curves (orange and blue) start in a deliberately bad guess. Press Step to run one round of EM, or Run to let it iterate.

<p class="hint">These points secretly come from <b>two overlapping clusters</b>, but the labels are hidden. The orange and blue curves start with a bad guess. Each <b>Step</b> runs one round of EM: shade every point by which curve likely produced it (E-step), then recompute each curve's center and width (M-step). The log-likelihood at the top never goes down.</p>
<div class="ll">log-likelihood: <span id="ll">—</span> &nbsp;·&nbsp; iteration <span id="it">0</span></div>
<canvas id="cv" width="520" height="220"></canvas>
<div class="btns">
  <button id="step" type="button">Step</button>
  <button id="run" type="button">Run</button>
  <button id="reset" type="button" class="ghost">Reset</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
.ll { font: 700 14px ui-monospace, monospace; color: #1d3557; margin: .2rem 0 .5rem; }
canvas { width: 100%; max-width: 520px; height: auto; background: #f5f7fa;
         border: 1px solid #cdd9e3; border-radius: 8px; display: block; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin-top: .6rem; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
button:disabled { opacity: .5; cursor: default; }

// 1-D Gaussian mixture fit by Expectation-Maximization.
// Two hidden clusters generated the data; EM recovers their means and widths
// by alternating: E-step = responsibility of each curve for each point,
// M-step = re-fit mean/variance/weight as a weighted average. The data
// log-likelihood provably never decreases (Dempster, Laird & Rubin, 1977).
const cv = document.getElementById('cv'), ctx = cv.getContext('2d');
const W = cv.width, H = cv.height, X0 = 0, X1 = 100;
const ORANGE = '#e8853a', BLUE = '#1d6fb8';

// Deterministic "random" points from two true clusters (~30 and ~70).
function makeData(){
  let s = 12345; const rnd = () => (s = (s*1103515245 + 12345) & 0x7fffffff) / 0x7fffffff;
  const gauss = (m,sd) => { let u=0,v=0; while(!u)u=rnd(); while(!v)v=rnd();
    return m + sd*Math.sqrt(-2*Math.log(u))*Math.cos(2*Math.PI*v); };
  const pts=[]; for(let i=0;i<30;i++) pts.push(gauss(32,7));
  for(let i=0;i<30;i++) pts.push(gauss(68,8));
  return pts.map(x=>Math.max(X0+2,Math.min(X1-2,x)));
}
let data = makeData();
// Two components: {mu, sigma, w}. Deliberately bad start (both near center).
let comp, iter, lastLL;
function init(){
  comp = [ {mu:45, sigma:10, w:0.5}, {mu:55, sigma:10, w:0.5} ];
  iter = 0; lastLL = null; updateLabels(null); draw(null);
}
const norm = (x,m,sd) => Math.exp(-((x-m)*(x-m))/(2*sd*sd)) / (Math.sqrt(2*Math.PI)*sd);

// One EM iteration. Returns the responsibilities used (for shading) + log-likelihood.
function emStep(){
  const n = data.length;
  const resp = data.map(()=>[0,0]);
  let ll = 0;
  // E-step: responsibility r_ik = w_k N(x|k) / sum_k ...
  for(let i=0;i<n;i++){
    const a = comp[0].w*norm(data[i],comp[0].mu,comp[0].sigma);
    const b = comp[1].w*norm(data[i],comp[1].mu,comp[1].sigma);
    const s = a+b || 1e-12;
    resp[i][0] = a/s; resp[i][1] = b/s;
    ll += Math.log(s);
  }
  // M-step: weighted re-fit of w, mu, sigma for each component.
  for(let k=0;k<2;k++){
    let nk=0, sum=0; for(let i=0;i<n;i++){ nk+=resp[i][k]; sum+=resp[i][k]*data[i]; }
    nk = nk || 1e-9;
    const mu = sum/nk;
    let varsum=0; for(let i=0;i<n;i++) varsum += resp[i][k]*(data[i]-mu)*(data[i]-mu);
    comp[k] = { mu, sigma: Math.max(2, Math.sqrt(varsum/nk)), w: nk/n };
  }
  iter++; lastLL = ll; updateLabels(ll); draw(resp);
  return ll;
}
function updateLabels(ll){
  document.getElementById('ll').textContent = ll===null ? '—' : ll.toFixed(2);
  document.getElementById('it').textContent = iter;
}
const sx = x => (x-X0)/(X1-X0)*W;
function draw(resp){
  ctx.clearRect(0,0,W,H);
  // curves (scaled so the taller peak fills most of the height)
  let peak = 0;
  for(let px=0;px<=W;px+=2){ const x = X0+(px/W)*(X1-X0);
    peak = Math.max(peak, comp[0].w*norm(x,comp[0].mu,comp[0].sigma)+comp[1].w*norm(x,comp[1].mu,comp[1].sigma)); }
  peak = peak || 1;
  const drawCurve = (k,color) => { ctx.strokeStyle=color; ctx.lineWidth=2.2; ctx.beginPath();
    for(let px=0;px<=W;px+=2){ const x=X0+(px/W)*(X1-X0);
      const y=H-20 - (comp[k].w*norm(x,comp[k].mu,comp[k].sigma)/peak)*(H-40);
      px===0?ctx.moveTo(px,y):ctx.lineTo(px,y); } ctx.stroke(); };
  drawCurve(0,ORANGE); drawCurve(1,BLUE);
  // points, shaded by responsibility (orange vs blue), on the baseline
  for(let i=0;i<data.length;i++){
    const r = resp ? resp[i][0] : 0.5;
    const col = mix(BLUE, ORANGE, r);
    ctx.fillStyle = col; ctx.beginPath();
    ctx.arc(sx(data[i]), H-12, 4, 0, 2*Math.PI); ctx.fill();
  }
  ctx.strokeStyle='#cdd9e3'; ctx.beginPath(); ctx.moveTo(0,H-6); ctx.lineTo(W,H-6); ctx.stroke();
}
function mix(c1,c2,t){ // t=1 -> c2
  const p=h=>[parseInt(h.slice(1,3),16),parseInt(h.slice(3,5),16),parseInt(h.slice(5,7),16)];
  const a=p(c1), b=p(c2); const r=a.map((v,i)=>Math.round(v+(b[i]-v)*t));
  return 'rgb('+r.join(',')+')';
}
let timer=null;
function stop(){ if(timer){ clearInterval(timer); timer=null;
  document.getElementById('run').textContent='Run'; } }
document.getElementById('step').addEventListener('click',()=>{ stop(); emStep();
  console.log('iter',iter,'logL',lastLL.toFixed(3),'means',comp[0].mu.toFixed(1),comp[1].mu.toFixed(1)); });
document.getElementById('run').addEventListener('click',function(){
  if(timer){ stop(); return; }
  this.textContent='Stop';
  timer=setInterval(()=>{ const prev=lastLL; emStep();
    if(prev!==null && Math.abs(lastLL-prev)<1e-4) stop(); }, 500);
});
document.getElementById('reset').addEventListener('click',()=>{ stop(); init(); });
init();
Result
Console

Each Step does two things. First the E-step: every point is shaded by how likely each curve is to have produced it. Then the M-step: each curve's center and width are recomputed as a weighted average over the points it "owns." Watch the log-likelihood number at the top — it never goes down. The two curves drift apart, settle over the two real clusters, and stop moving. That stopping point is a local optimum: change the starting guess and EM may land somewhere slightly different.

The Real Complexity

EM has an unusually clean guarantee for an iterative method, and an unusually honest limitation.

  • It never gets worse. Dempster, Laird and Rubin (1977) proved that each E-step / M-step pair can only increase (or hold) the data's likelihood. The numbers in the demo climb monotonically — that is not a coincidence, it is a theorem.
  • It converges — but only locally. Because the likelihood keeps rising and is bounded, EM settles down. The catch: it settles on a local optimum, not necessarily the global best. A poor starting guess can trap it on a mediocre fit, which is why practitioners restart from several random seeds.
  • One step is cheap. Each iteration is roughly linear in (points × clusters), so EM scales gracefully to large datasets — the cost is the number of iterations, not any single one.
  • The global problem is hard. Finding the single best mixture that maximizes likelihood is NP-hard in general. EM is the practical compromise: fast, reliable improvement with no promise of perfection.

So EM is not a magic solver. It is a hill-climber for a landscape that is genuinely rugged — closely related to the hardness of clustering itself, the same wall you meet in k-means.

Where It Matters

"I have data, but part of it is hidden" describes an enormous slice of real statistics, and EM is the default tool for it:

  • Clustering and segmentation: Gaussian mixture models group customers, pixels or sensor readings without any labels — the soft cousin of k-means.
  • Speech and language: training the hidden Markov models behind classic speech recognition uses EM (the Baum-Welch algorithm) to infer the unseen state sequence.
  • Genetics and biology: EM untangles overlapping signals in gene-expression and population-genetics data where the underlying groups are unknown.
  • Topic models and recommendations: latent factors behind documents or ratings are exactly the "hidden variables" EM was built to estimate, close in spirit to Bayesian inference.

Whenever the thing you most want to know is the thing you can't observe, EM gives you a principled way to bootstrap it from the data you can see.

Conclusion

Expectation-Maximization is a small idea with a long reach: when you can't see the labels, invent them, refit the model, and let the two halves pull each other into focus. Each round provably improves the fit, and the procedure stops at a stable, sensible answer.

The honest fine print is that the answer is only locally best — the global optimum stays NP-hard, so a bad start can mislead it. But for countless problems where half the data is missing, EM is the dependable, elegant workhorse that turns a paradox into a picture. The next time a model seems impossible to fit because the key information is hidden, remember: guess, refit, repeat.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/expectation-maximization/Content licensed under CC BY-NC 4.0.