Opening the Black Box: Mechanistic Interpretability

Introduction

Modern neural networks are extraordinary pattern machines. Train one on enough data and it learns to translate languages, generate images, or write code. What it does not do is explain itself.

Mechanistic interpretability is the project of reverse-engineering neural networks the way a hardware engineer reverse-engineers a chip — not just confirming that the thing works, but tracing exactly which computation each part performs. The goal is to open the black box completely: to read the wiring diagram.

The idea is older than deep learning, but the field accelerated dramatically after 2020 when researchers at Anthropic and elsewhere began finding that real circuits — not metaphors — exist inside trained models. A specific curve detector in a vision model is not one neuron but a precise loop of neurons, each responding to a rotated version of the edge. Understanding those circuits is now seen as essential to building AI systems we can actually trust.

This sits at the intersection of computer science, neuroscience and mathematics. It is not yet a solved science. But the early results are striking enough that "open" is the honest status — we have no general algorithm for interpreting arbitrary networks, and no proof that a complete interpretation is always possible.

See What Each Neuron Detects

Below is a tiny two-layer neural network (4 hidden neurons) trained on 2D points colored red or blue. Each hidden neuron learns a linear boundary — a direction in the input space it responds to. Together they compose into the full decision.

Click Train to fit the network, then click any neuron button to highlight its activation pattern across the plane. The brighter a region, the more strongly that neuron fires there. Notice how the four neurons carve up the space in complementary ways — that division of labour is the circuit at work.

<p class="hint">A 2-input → 4-hidden → 1-output network trained on 2D data (red vs blue,
    each split into two clusters). Click <b>Train</b>, then click a <b>Neuron N</b> button to
    highlight where that neuron fires most strongly. Each neuron learns a different region of
    the plane; together they form the full decision circuit.</p>
<div class="controls">
  <button id="btnTrain">Train</button>
  <button id="btnN0" class="nbtn" disabled>Neuron 1</button>
  <button id="btnN1" class="nbtn" disabled>Neuron 2</button>
  <button id="btnN2" class="nbtn" disabled>Neuron 3</button>
  <button id="btnN3" class="nbtn" disabled>Neuron 4</button>
  <button id="btnDec" class="nbtn" disabled>Decision</button>
</div>
<div class="canvas-wrap">
  <canvas id="c" width="340" height="290"></canvas>
  <canvas id="ov" width="340" height="290"></canvas>
</div>
<div id="info" class="info">Click Train to fit the network, then explore each neuron.</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; background: #f7f9fb; }
.hint { font-size: .87rem; color: #444; margin: 0 0 .6rem; line-height: 1.5; }
.controls { display: flex; flex-wrap: wrap; gap: .4rem; margin-bottom: .6rem; }
button { font: 600 13px system-ui; padding: .38rem .8rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; }
button:disabled { opacity: .38; cursor: default; }
button.nbtn { background: #fff; color: #1d3557; }
button.nbtn.on { background: #1d3557; color: #fff; }
.canvas-wrap { position: relative; display: inline-block; border-radius: 10px; overflow: hidden;
               box-shadow: 0 2px 8px #0001; }
#c, #ov { display: block; }
#ov { position: absolute; top: 0; left: 0; pointer-events: none; }
.info { margin-top: .5rem; font-size: .84rem; color: #1d3557; min-height: 1.4em; font-weight: 600; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Mechanistic interpretability demo:
// MLP (2→4 ReLU →1 sigmoid) trained on 4 Gaussian clusters (2 red, 2 blue).
// Best-of-3 restarts used to reliably avoid dead-neuron traps.
// After training, click each neuron button to see its activation heatmap.

const CW = 340, CH = 290;
const canvas = document.getElementById('c'), ctx = canvas.getContext('2d');
const ol = document.getElementById('ov'), octx = ol.getContext('2d');
const info = document.getElementById('info');

// ── Dataset: 4 Gaussian clusters ─────────────────────────────────────────
function gauss(mu, sigma, n, label) {
  const out = [];
  for (let i = 0; i < n; i++) {
    // Box-Muller transform for normal distribution
    const u = Math.random(), v = Math.random();
    const z0 = Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v);
    const z1 = Math.sqrt(-2 * Math.log(u)) * Math.sin(2 * Math.PI * v);
    out.push({ x: mu[0] + sigma * z0, y: mu[1] + sigma * z1, label });
  }
  return out;
}
function makeData() {
  return [
    ...gauss([ 0.38,  0.32], 0.12, 35, 0), // red  top-right
    ...gauss([-0.38, -0.32], 0.12, 35, 0), // red  bottom-left
    ...gauss([-0.38,  0.32], 0.12, 35, 1), // blue top-left
    ...gauss([ 0.38, -0.32], 0.12, 35, 1), // blue bottom-right
  ];
}
let data = makeData();

// ── Network state (global, replaced after training) ───────────────────────
let W1 = Array.from({length:4}, () => [0,0]);
let b1 = [0,0,0,0];
let W2 = [0,0,0,0];
let b2 = 0;

const relu = x => x > 0 ? x : 0;
const sig  = x => 1 / (1 + Math.exp(-x));

function fwd(x0, x1) {
  const h = W1.map((w, i) => relu(w[0]*x0 + w[1]*x1 + b1[i]));
  let z = b2; for (let i = 0; i < 4; i++) z += W2[i] * h[i];
  return { h, out: sig(z) };
}
function neuronAct(x0, x1, ni) {
  return relu(W1[ni][0]*x0 + W1[ni][1]*x1 + b1[ni]);
}

// ── Training (single run, returns weights + loss) ────────────────────────
function trainOnce(dat) {
  const lW1 = Array.from({length:4}, () => [(Math.random()-.5)*1.0, (Math.random()-.5)*1.0]);
  const lb1 = [0.1, 0.1, 0.1, 0.1]; // small positive bias prevents dead neurons
  const lW2 = [(Math.random()-.5)*1.0, (Math.random()-.5)*1.0,
               (Math.random()-.5)*1.0, (Math.random()-.5)*1.0];
  let lb2 = 0;

  function lfwd(x0, x1) {
    const h = lW1.map((w, i) => relu(w[0]*x0 + w[1]*x1 + lb1[i]));
    let z = lb2; for (let i=0;i<4;i++) z+=lW2[i]*h[i]; return {h, out:sig(z)};
  }

  const lr = 0.1;
  const arr = dat.slice();
  for (let ep = 0; ep < 1400; ep++) {
    for (let i=arr.length-1;i>0;i--){const j=Math.floor(Math.random()*(i+1));[arr[i],arr[j]]=[arr[j],arr[i]];}
    for (const p of arr) {
      const {h, out} = lfwd(p.x, p.y);
      const dO = out - p.label;
      const dH = lW2.map((w, i) => dO * w * (h[i] > 0 ? 1 : 0));
      for (let i=0;i<4;i++){
        lW1[i][0]-=lr*dH[i]*p.x; lW1[i][1]-=lr*dH[i]*p.y;
        lb1[i]-=lr*dH[i]; lW2[i]-=lr*dO*h[i];
      }
      lb2 -= lr * dO;
    }
  }
  // Compute BCE loss for restart selection
  let loss = 0;
  for (const p of dat) {
    const {out} = lfwd(p.x, p.y);
    loss -= p.label*Math.log(out+1e-9) + (1-p.label)*Math.log(1-out+1e-9);
  }
  return {W1:lW1, b1:lb1, W2:lW2, b2:lb2, loss};
}

// ── Train with best-of-3 restarts ─────────────────────────────────────────
function trainNet() {
  let best = null;
  for (let r = 0; r < 3; r++) {
    const res = trainOnce(data);
    if (!best || res.loss < best.loss) best = res;
  }
  // Install best weights globally
  W1 = best.W1; b1 = best.b1; W2 = best.W2; b2 = best.b2;
}

// ── Coordinate transforms ─────────────────────────────────────────────
const PAD = 18, RANGE = 1.1;
const toC   = (x,y)  => [PAD+((x+RANGE/2)/RANGE)*(CW-2*PAD), PAD+((RANGE/2-y)/RANGE)*(CH-2*PAD)];
const fromC = (cx,cy) => [(cx-PAD)/(CW-2*PAD)*RANGE-RANGE/2, RANGE/2-(cy-PAD)/(CH-2*PAD)*RANGE];

// ── Heatmap rendering ────────────────────────────────────────────────
function drawHeatmap(ni) {
  const idata = octx.createImageData(CW, CH);
  let maxA = 0;
  if (ni >= 0) {
    for (let px=0;px<CW;px+=4) for (let py=0;py<CH;py+=4) {
      const [x,y]=fromC(px+2,py+2), a=neuronAct(x,y,ni); if(a>maxA) maxA=a;
    }
    if (maxA === 0) maxA = 1;
  }
  for (let px=0;px<CW;px++) for (let py=0;py<CH;py++) {
    const [x,y] = fromC(px,py);
    let r, g, b;
    if (ni >= 0) {
      const a = neuronAct(x, y, ni) / maxA;
      // Warm amber gradient: white → deep orange
      r = 255; g = Math.round(250 - a*110); b = Math.round(240 - a*240);
    } else {
      const p = fwd(x, y).out;
      const s = Math.abs(p - 0.5) * 2;
      if (p > 0.5) { r=Math.round(220-s*140); g=Math.round(230-s*120); b=Math.round(240+s*15); }
      else         { r=Math.round(240+s*15);  g=Math.round(220-s*120); b=Math.round(220-s*140); }
    }
    const idx = 4*(py*CW+px);
    idata.data[idx]=r; idata.data[idx+1]=g; idata.data[idx+2]=b; idata.data[idx+3]=190;
  }
  octx.putImageData(idata, 0, 0);
}

// ── Draw data points ──────────────────────────────────────────────────────
function drawData() {
  ctx.clearRect(0,0,CW,CH);
  ctx.fillStyle='#f7f9fb'; ctx.fillRect(0,0,CW,CH);
  ctx.strokeStyle='#d0d8e0'; ctx.lineWidth=.8;
  const [ox,oy]=toC(0,0);
  ctx.beginPath(); ctx.moveTo(ox,PAD); ctx.lineTo(ox,CH-PAD); ctx.stroke();
  ctx.beginPath(); ctx.moveTo(PAD,oy); ctx.lineTo(CW-PAD,oy); ctx.stroke();
  for (const p of data) {
    const [cx,cy]=toC(p.x,p.y);
    ctx.beginPath(); ctx.arc(cx,cy,4,0,2*Math.PI);
    ctx.fillStyle=p.label===0?'#e63946':'#1d78d4';
    ctx.strokeStyle='#fff'; ctx.lineWidth=1.2;
    ctx.fill(); ctx.stroke();
  }
}

// ── Button logic ──────────────────────────────────────────────────────────
let activeBtn = null;
function setActive(btn, ni) {
  if (activeBtn) activeBtn.classList.remove('on');
  btn.classList.add('on'); activeBtn = btn;
  drawData(); drawHeatmap(ni);
  if (ni >= 0) {
    const w = W1[ni];
    const xDir = w[0] > 0 ? '+x' : '−x';
    const yDir = w[1] > 0 ? '+y' : '−y';
    info.textContent = fmt('Neuron %ni%%v%: weights ((%v2%, %v3%)), bias %v4% — responds to %xDir%, %yDir% direction', { ni: ni, v: 1, v2: w[0].toFixed(3), v3: w[1].toFixed(3), v4: b1[ni].toFixed(3), xDir: xDir, yDir: yDir });
  } else {
    info.textContent = 'Decision: all 4 neurons vote → red (class 0) or blue (class 1).';
  }
}

document.getElementById('btnTrain').addEventListener('click', () => {
  info.textContent = 'Training (3 restarts, picking best)…';
  setTimeout(() => {
    trainNet();
    let correct = 0;
    for (const p of data) { if ((fwd(p.x,p.y).out>.5?1:0)===p.label) correct++; }
    const acc = (correct/data.length*100).toFixed(1);
    ['btnN0','btnN1','btnN2','btnN3','btnDec'].forEach(id=>document.getElementById(id).disabled=false);
    drawData(); octx.clearRect(0,0,CW,CH);
    info.textContent = fmt('Trained! Accuracy: %acc%%. Click a neuron button to see what it detects.', { acc: acc });
  }, 30);
});

document.getElementById('btnN0').addEventListener('click', function(){ setActive(this, 0); });
document.getElementById('btnN1').addEventListener('click', function(){ setActive(this, 1); });
document.getElementById('btnN2').addEventListener('click', function(){ setActive(this, 2); });
document.getElementById('btnN3').addEventListener('click', function(){ setActive(this, 3); });
document.getElementById('btnDec').addEventListener('click', function(){ setActive(this, -1); });

drawData();

<p class="hint">A 2-input → 4-hidden → 1-output network trained on 2D data (red vs blue,
    each split into two clusters). Click <b>Train</b>, then click a <b>Neuron N</b> button to
    highlight where that neuron fires most strongly. Each neuron learns a different region of
    the plane; together they form the full decision circuit.</p>
<div class="controls">
  <button id="btnTrain">Train</button>
  <button id="btnN0" class="nbtn" disabled>Neuron 1</button>
  <button id="btnN1" class="nbtn" disabled>Neuron 2</button>
  <button id="btnN2" class="nbtn" disabled>Neuron 3</button>
  <button id="btnN3" class="nbtn" disabled>Neuron 4</button>
  <button id="btnDec" class="nbtn" disabled>Decision</button>
</div>
<div class="canvas-wrap">
  <canvas id="c" width="340" height="290"></canvas>
  <canvas id="ov" width="340" height="290"></canvas>
</div>
<div id="info" class="info">Click Train to fit the network, then explore each neuron.</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; background: #f7f9fb; }
.hint { font-size: .87rem; color: #444; margin: 0 0 .6rem; line-height: 1.5; }
.controls { display: flex; flex-wrap: wrap; gap: .4rem; margin-bottom: .6rem; }
button { font: 600 13px system-ui; padding: .38rem .8rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; }
button:disabled { opacity: .38; cursor: default; }
button.nbtn { background: #fff; color: #1d3557; }
button.nbtn.on { background: #1d3557; color: #fff; }
.canvas-wrap { position: relative; display: inline-block; border-radius: 10px; overflow: hidden;
               box-shadow: 0 2px 8px #0001; }
#c, #ov { display: block; }
#ov { position: absolute; top: 0; left: 0; pointer-events: none; }
.info { margin-top: .5rem; font-size: .84rem; color: #1d3557; min-height: 1.4em; font-weight: 600; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Mechanistic interpretability demo:
// MLP (2→4 ReLU →1 sigmoid) trained on 4 Gaussian clusters (2 red, 2 blue).
// Best-of-3 restarts used to reliably avoid dead-neuron traps.
// After training, click each neuron button to see its activation heatmap.

const CW = 340, CH = 290;
const canvas = document.getElementById('c'), ctx = canvas.getContext('2d');
const ol = document.getElementById('ov'), octx = ol.getContext('2d');
const info = document.getElementById('info');

// ── Dataset: 4 Gaussian clusters ─────────────────────────────────────────
function gauss(mu, sigma, n, label) {
  const out = [];
  for (let i = 0; i < n; i++) {
    // Box-Muller transform for normal distribution
    const u = Math.random(), v = Math.random();
    const z0 = Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v);
    const z1 = Math.sqrt(-2 * Math.log(u)) * Math.sin(2 * Math.PI * v);
    out.push({ x: mu[0] + sigma * z0, y: mu[1] + sigma * z1, label });
  }
  return out;
}
function makeData() {
  return [
    ...gauss([ 0.38,  0.32], 0.12, 35, 0), // red  top-right
    ...gauss([-0.38, -0.32], 0.12, 35, 0), // red  bottom-left
    ...gauss([-0.38,  0.32], 0.12, 35, 1), // blue top-left
    ...gauss([ 0.38, -0.32], 0.12, 35, 1), // blue bottom-right
  ];
}
let data = makeData();

// ── Network state (global, replaced after training) ───────────────────────
let W1 = Array.from({length:4}, () => [0,0]);
let b1 = [0,0,0,0];
let W2 = [0,0,0,0];
let b2 = 0;

const relu = x => x > 0 ? x : 0;
const sig  = x => 1 / (1 + Math.exp(-x));

function fwd(x0, x1) {
  const h = W1.map((w, i) => relu(w[0]*x0 + w[1]*x1 + b1[i]));
  let z = b2; for (let i = 0; i < 4; i++) z += W2[i] * h[i];
  return { h, out: sig(z) };
}
function neuronAct(x0, x1, ni) {
  return relu(W1[ni][0]*x0 + W1[ni][1]*x1 + b1[ni]);
}

// ── Training (single run, returns weights + loss) ────────────────────────
function trainOnce(dat) {
  const lW1 = Array.from({length:4}, () => [(Math.random()-.5)*1.0, (Math.random()-.5)*1.0]);
  const lb1 = [0.1, 0.1, 0.1, 0.1]; // small positive bias prevents dead neurons
  const lW2 = [(Math.random()-.5)*1.0, (Math.random()-.5)*1.0,
               (Math.random()-.5)*1.0, (Math.random()-.5)*1.0];
  let lb2 = 0;

  function lfwd(x0, x1) {
    const h = lW1.map((w, i) => relu(w[0]*x0 + w[1]*x1 + lb1[i]));
    let z = lb2; for (let i=0;i<4;i++) z+=lW2[i]*h[i]; return {h, out:sig(z)};
  }

  const lr = 0.1;
  const arr = dat.slice();
  for (let ep = 0; ep < 1400; ep++) {
    for (let i=arr.length-1;i>0;i--){const j=Math.floor(Math.random()*(i+1));[arr[i],arr[j]]=[arr[j],arr[i]];}
    for (const p of arr) {
      const {h, out} = lfwd(p.x, p.y);
      const dO = out - p.label;
      const dH = lW2.map((w, i) => dO * w * (h[i] > 0 ? 1 : 0));
      for (let i=0;i<4;i++){
        lW1[i][0]-=lr*dH[i]*p.x; lW1[i][1]-=lr*dH[i]*p.y;
        lb1[i]-=lr*dH[i]; lW2[i]-=lr*dO*h[i];
      }
      lb2 -= lr * dO;
    }
  }
  // Compute BCE loss for restart selection
  let loss = 0;
  for (const p of dat) {
    const {out} = lfwd(p.x, p.y);
    loss -= p.label*Math.log(out+1e-9) + (1-p.label)*Math.log(1-out+1e-9);
  }
  return {W1:lW1, b1:lb1, W2:lW2, b2:lb2, loss};
}

// ── Train with best-of-3 restarts ─────────────────────────────────────────
function trainNet() {
  let best = null;
  for (let r = 0; r < 3; r++) {
    const res = trainOnce(data);
    if (!best || res.loss < best.loss) best = res;
  }
  // Install best weights globally
  W1 = best.W1; b1 = best.b1; W2 = best.W2; b2 = best.b2;
}

// ── Coordinate transforms ─────────────────────────────────────────────
const PAD = 18, RANGE = 1.1;
const toC   = (x,y)  => [PAD+((x+RANGE/2)/RANGE)*(CW-2*PAD), PAD+((RANGE/2-y)/RANGE)*(CH-2*PAD)];
const fromC = (cx,cy) => [(cx-PAD)/(CW-2*PAD)*RANGE-RANGE/2, RANGE/2-(cy-PAD)/(CH-2*PAD)*RANGE];

// ── Heatmap rendering ────────────────────────────────────────────────
function drawHeatmap(ni) {
  const idata = octx.createImageData(CW, CH);
  let maxA = 0;
  if (ni >= 0) {
    for (let px=0;px<CW;px+=4) for (let py=0;py<CH;py+=4) {
      const [x,y]=fromC(px+2,py+2), a=neuronAct(x,y,ni); if(a>maxA) maxA=a;
    }
    if (maxA === 0) maxA = 1;
  }
  for (let px=0;px<CW;px++) for (let py=0;py<CH;py++) {
    const [x,y] = fromC(px,py);
    let r, g, b;
    if (ni >= 0) {
      const a = neuronAct(x, y, ni) / maxA;
      // Warm amber gradient: white → deep orange
      r = 255; g = Math.round(250 - a*110); b = Math.round(240 - a*240);
    } else {
      const p = fwd(x, y).out;
      const s = Math.abs(p - 0.5) * 2;
      if (p > 0.5) { r=Math.round(220-s*140); g=Math.round(230-s*120); b=Math.round(240+s*15); }
      else         { r=Math.round(240+s*15);  g=Math.round(220-s*120); b=Math.round(220-s*140); }
    }
    const idx = 4*(py*CW+px);
    idata.data[idx]=r; idata.data[idx+1]=g; idata.data[idx+2]=b; idata.data[idx+3]=190;
  }
  octx.putImageData(idata, 0, 0);
}

// ── Draw data points ──────────────────────────────────────────────────────
function drawData() {
  ctx.clearRect(0,0,CW,CH);
  ctx.fillStyle='#f7f9fb'; ctx.fillRect(0,0,CW,CH);
  ctx.strokeStyle='#d0d8e0'; ctx.lineWidth=.8;
  const [ox,oy]=toC(0,0);
  ctx.beginPath(); ctx.moveTo(ox,PAD); ctx.lineTo(ox,CH-PAD); ctx.stroke();
  ctx.beginPath(); ctx.moveTo(PAD,oy); ctx.lineTo(CW-PAD,oy); ctx.stroke();
  for (const p of data) {
    const [cx,cy]=toC(p.x,p.y);
    ctx.beginPath(); ctx.arc(cx,cy,4,0,2*Math.PI);
    ctx.fillStyle=p.label===0?'#e63946':'#1d78d4';
    ctx.strokeStyle='#fff'; ctx.lineWidth=1.2;
    ctx.fill(); ctx.stroke();
  }
}

// ── Button logic ──────────────────────────────────────────────────────────
let activeBtn = null;
function setActive(btn, ni) {
  if (activeBtn) activeBtn.classList.remove('on');
  btn.classList.add('on'); activeBtn = btn;
  drawData(); drawHeatmap(ni);
  if (ni >= 0) {
    const w = W1[ni];
    const xDir = w[0] > 0 ? '+x' : '−x';
    const yDir = w[1] > 0 ? '+y' : '−y';
    info.textContent = fmt('Neuron %ni%%v%: weights ((%v2%, %v3%)), bias %v4% — responds to %xDir%, %yDir% direction', { ni: ni, v: 1, v2: w[0].toFixed(3), v3: w[1].toFixed(3), v4: b1[ni].toFixed(3), xDir: xDir, yDir: yDir });
  } else {
    info.textContent = 'Decision: all 4 neurons vote → red (class 0) or blue (class 1).';
  }
}

document.getElementById('btnTrain').addEventListener('click', () => {
  info.textContent = 'Training (3 restarts, picking best)…';
  setTimeout(() => {
    trainNet();
    let correct = 0;
    for (const p of data) { if ((fwd(p.x,p.y).out>.5?1:0)===p.label) correct++; }
    const acc = (correct/data.length*100).toFixed(1);
    ['btnN0','btnN1','btnN2','btnN3','btnDec'].forEach(id=>document.getElementById(id).disabled=false);
    drawData(); octx.clearRect(0,0,CW,CH);
    info.textContent = fmt('Trained! Accuracy: %acc%%. Click a neuron button to see what it detects.', { acc: acc });
  }, 30);
});

document.getElementById('btnN0').addEventListener('click', function(){ setActive(this, 0); });
document.getElementById('btnN1').addEventListener('click', function(){ setActive(this, 1); });
document.getElementById('btnN2').addEventListener('click', function(){ setActive(this, 2); });
document.getElementById('btnN3').addEventListener('click', function(){ setActive(this, 3); });
document.getElementById('btnDec').addEventListener('click', function(){ setActive(this, -1); });

drawData();

Result

Console

The key insight: checking that a neuron detects a feature (just read its weights) is easy. Finding a human-interpretable description of what every neuron does in a large network is the unsolved problem. With four neurons and two inputs it is almost trivial; with billions of parameters it becomes a research frontier.

The Real Complexity

Reading a single neuron's weights is instant. Understanding what a whole network computes is a different matter.

Superposition: a network with n neurons can represent far more than n distinct features simultaneously by packing them into overlapping directions in activation space. This was formalized by Ely Hadley, Chris Olah et al. at Anthropic (2022). A single neuron is almost never responsible for just one concept.
Polysemanticity: as a direct consequence of superposition, individual neurons often respond to multiple unrelated features — a neuron in a language model might fire for "banana", "curve" and a syntactic pattern all at once. Identifying which "meaning" it carries in any given context requires tracking the full circuit, not just the neuron.
Combinatorial explosion: a network with n neurons has $O(2^{n})$ possible sub-circuits. There is no known polynomial-time algorithm that, given a trained network, produces a faithful human-readable description of it. Whether one can exist is open.
Formal hardness: while no tight completeness result pins down interpretability in the NP or PSPACE hierarchy (the question is still being formulated), the search for circuits resembles constraint satisfaction — related problems are NP-hard under natural formulations.

The status of the field: open. Researchers have reverse-engineered specific circuits in small models (curve detectors, induction heads, modular addition) and even found that large models store factual associations in identifiable "knowledge neurons." But a general theory of network interpretation — applicable to arbitrary architectures at arbitrary scale — does not yet exist. This is why mechanistic interpretability is one of the most active research programs in AI safety today, pursued heavily at Anthropic, DeepMind, and several universities.

A related puzzle is that interpretability may have fundamental limits: if a network has stored information in superposition across exponentially many directions, a full account of it might require exponentially long descriptions. Whether compression is always possible is unknown.

Where It Matters

Understanding what a neural network actually computes is not just an intellectual puzzle — it has direct consequences:

AI safety: if we can verify that a model's internal circuits do not contain deceptive goals or hidden optimization targets, we have a much stronger safety argument than any behavioral test alone. This is the primary motivation for interpretability work at Anthropic.
Debugging and fixing failures: when a model makes a systematic error, mechanistic interpretability can locate the responsible circuit and patch it — a surgical edit rather than retraining from scratch. The technique of "activation patching" can identify exactly which attention head or MLP layer encodes a wrong fact.
Knowledge editing: research on "knowledge neurons" (Dai et al., 2021) showed that specific neurons in large language models encode factual associations, enabling targeted edits — changing "Eiffel Tower → Paris" without disturbing unrelated knowledge.
Scientific discovery: training neural networks on physics or chemistry data and then reading their circuits has begun to reveal compact mathematical laws the networks have independently discovered, connecting interpretability to the philosophy of science.
Model auditing: regulators increasingly want explanations for AI decisions. Mechanistic interpretability is the only approach that can, in principle, provide a causal account rather than a post-hoc rationalization.

For more on how algorithms are represented inside learned models, see neural network training and PAC learning.

Conclusion

Mechanistic interpretability asks a deceptively simple question: what is this neuron doing? The answer, it turns out, requires understanding superposition, polysemanticity, and the full compositional structure of a network — none of which yields easily to any known general algorithm.

The field has already produced genuine surprises: real circuits for curve detection, induction, and modular arithmetic found inside trained models; knowledge stored in identifiable neurons; mathematical laws rediscovered by networks trained on physics data. Each result is a proof of concept that the black box can, at least in small patches, be opened.

What remains open is whether the whole box can be opened — whether every computation a large network performs can be expressed in a form a human can check and trust. That question is inseparable from the question of whether we can ever deploy AI systems with genuine confidence rather than hopeful approximation. The answer will define the next decade of the field.

Comments

Loading comments...

https://www.kipuhub.com/en/article/mechanistic-interpretability/Content licensed under CC BY-NC 4.0.

Cookies & privacy

Mechanistic Interpretability

What is each neuron actually doing?