Mechanistic Interpretability

Modern neural networks are extraordinary pattern machines. Train one on enough data and it learns to translate languages, generate images, or write code. What it does not do is explain itself.

Mechanistic interpretability is the project of reverse-engineering neural networks the way a hardware engineer reverse-engineers a chip — not just confirming that the thing works, but tracing exactly which computation each part performs. The goal is to open the black box completely: to read the wiring diagram.

The idea is older than deep learning, but the field accelerated dramatically after 2020 when researchers at Anthropic and elsewhere began finding that real circuits — not metaphors — exist inside trained models. A specific curve detector in a vision model is not one neuron but a precise loop of neurons, each responding to a rotated version of the edge. Understanding those circuits is now seen as essential to building AI systems we can actually trust.

This sits at the intersection of computer science, neuroscience and mathematics. It is not yet a solved science. But the early results are striking enough that "open" is the honest status — we have no general algorithm for interpreting arbitrary networks, and no proof that a complete interpretation is always possible.

Below is a tiny two-layer neural network (4 hidden neurons) trained on 2D points colored red or blue. Each hidden neuron learns a linear boundary — a direction in the input space it responds to. Together they compose into the full decision.

Click Train to fit the network, then click any neuron button to highlight its activation pattern across the plane. The brighter a region, the more strongly that neuron fires there. Notice how the four neurons carve up the space in complementary ways — that division of labour is the circuit at work.

<p class="hint">{{hint}}</p>
<div class="controls">
  <button id="btnTrain">{{btn_train}}</button>
  <button id="btnN0" class="nbtn" disabled>{{btn_neuron1}}</button>
  <button id="btnN1" class="nbtn" disabled>{{btn_neuron2}}</button>
  <button id="btnN2" class="nbtn" disabled>{{btn_neuron3}}</button>
  <button id="btnN3" class="nbtn" disabled>{{btn_neuron4}}</button>
  <button id="btnDec" class="nbtn" disabled>{{btn_decision}}</button>
</div>
<div class="canvas-wrap">
  <canvas id="c" width="340" height="290"></canvas>
  <canvas id="ov" width="340" height="290"></canvas>
</div>
<div id="info" class="info">{{info_initial}}</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; background: #f7f9fb; }
.hint { font-size: .87rem; color: #444; margin: 0 0 .6rem; line-height: 1.5; }
.controls { display: flex; flex-wrap: wrap; gap: .4rem; margin-bottom: .6rem; }
button { font: 600 13px system-ui; padding: .38rem .8rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; }
button:disabled { opacity: .38; cursor: default; }
button.nbtn { background: #fff; color: #1d3557; }
button.nbtn.on { background: #1d3557; color: #fff; }
.canvas-wrap { position: relative; display: inline-block; border-radius: 10px; overflow: hidden;
               box-shadow: 0 2px 8px #0001; }
#c, #ov { display: block; }
#ov { position: absolute; top: 0; left: 0; pointer-events: none; }
.info { margin-top: .5rem; font-size: .84rem; color: #1d3557; min-height: 1.4em; font-weight: 600; }

// Code not found

The key insight: checking that a neuron detects a feature (just read its weights) is easy. Finding a human-interpretable description of what every neuron does in a large network is the unsolved problem. With four neurons and two inputs it is almost trivial; with billions of parameters it becomes a research frontier.

Reading a single neuron's weights is instant. Understanding what a whole network computes is a different matter.

• Superposition: a network with n neurons can represent far more than n distinct features simultaneously by packing them into overlapping directions in activation space. This was formalized by Ely Hadley, Chris Olah et al. at Anthropic (2022). A single neuron is almost never responsible for just one concept.
• Polysemanticity: as a direct consequence of superposition, individual neurons often respond to multiple unrelated features — a neuron in a language model might fire for "banana", "curve" and a syntactic pattern all at once. Identifying which "meaning" it carries in any given context requires tracking the full circuit, not just the neuron.
• Combinatorial explosion: a network with n neurons has $O(2^{n})$ possible sub-circuits. There is no known polynomial-time algorithm that, given a trained network, produces a faithful human-readable description of it. Whether one can exist is open.
• Formal hardness: while no tight completeness result pins down interpretability in the NP or PSPACE hierarchy (the question is still being formulated), the search for circuits resembles constraint satisfaction — related problems are NP-hard under natural formulations.

The status of the field: open. Researchers have reverse-engineered specific circuits in small models (curve detectors, induction heads, modular addition) and even found that large models store factual associations in identifiable "knowledge neurons." But a general theory of network interpretation — applicable to arbitrary architectures at arbitrary scale — does not yet exist. This is why mechanistic interpretability is one of the most active research programs in AI safety today, pursued heavily at Anthropic, DeepMind, and several universities.

A related puzzle is that interpretability may have fundamental limits: if a network has stored information in superposition across exponentially many directions, a full account of it might require exponentially long descriptions. Whether compression is always possible is unknown.

Understanding what a neural network actually computes is not just an intellectual puzzle — it has direct consequences:

• AI safety: if we can verify that a model's internal circuits do not contain deceptive goals or hidden optimization targets, we have a much stronger safety argument than any behavioral test alone. This is the primary motivation for interpretability work at Anthropic.
• Debugging and fixing failures: when a model makes a systematic error, mechanistic interpretability can locate the responsible circuit and patch it — a surgical edit rather than retraining from scratch. The technique of "activation patching" can identify exactly which attention head or MLP layer encodes a wrong fact.
• Knowledge editing: research on "knowledge neurons" (Dai et al., 2021) showed that specific neurons in large language models encode factual associations, enabling targeted edits — changing "Eiffel Tower → Paris" without disturbing unrelated knowledge.
• Scientific discovery: training neural networks on physics or chemistry data and then reading their circuits has begun to reveal compact mathematical laws the networks have independently discovered, connecting interpretability to the philosophy of science.
• Model auditing: regulators increasingly want explanations for AI decisions. Mechanistic interpretability is the only approach that can, in principle, provide a causal account rather than a post-hoc rationalization.

For more on how algorithms are represented inside learned models, see neural network training and PAC learning.

Mechanistic interpretability asks a deceptively simple question: what is this neuron doing? The answer, it turns out, requires understanding superposition, polysemanticity, and the full compositional structure of a network — none of which yields easily to any known general algorithm.

The field has already produced genuine surprises: real circuits for curve detection, induction, and modular arithmetic found inside trained models; knowledge stored in identifiable neurons; mathematical laws rediscovered by networks trained on physics data. Each result is a proof of concept that the black box can, at least in small patches, be opened.

What remains open is whether the whole box can be opened — whether every computation a large network performs can be expressed in a form a human can check and trust. That question is inseparable from the question of whether we can ever deploy AI systems with genuine confidence rather than hopeful approximation. The answer will define the next decade of the field.