Introduction

Imagine spending months training a billion-parameter neural network to recognize images. It is brilliant — and completely unusable on a phone. Now imagine training a tiny fifty-parameter model directly on the same images. It learns well, but it misses something the giant had.

The giant does not just output a label. It outputs a probability distribution: "I'm 91 % sure this is a cat, 7 % sure it's a fox, and 1 % sure it's a dog." Those small, non-zero numbers on the wrong classes — the soft targets — carry an enormous amount of structural information about how the world is organized. A hard label like cat throws all of that away.

Knowledge distillation, introduced by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean in 2015, exploits this insight. Instead of training the small student model on hard labels, you train it to mimic the full distribution produced by the large teacher. The student learns not just the right answer, but the teacher's entire sense of how close the alternatives are.

The result is striking: a student trained on soft targets routinely outperforms the same architecture trained directly on the data — sometimes matching a teacher ten times its size.

Try It: Soft vs Hard Targets

This demo simulates a simple 3-class classification problem (Cat / Fox / Dog). The teacher is a confident model whose outputs have been softened with a temperature parameter. The student trains on either hard one-hot labels or the teacher's soft probabilities.

<div class="hint">
  <strong>3-class problem:</strong> Cat / Fox / Dog. The <em>teacher</em> knows the true class and outputs soft probabilities.
  Compare how a <em>student</em> learns from hard one-hot labels vs. the teacher's soft distribution.
</div>
<div id="scenario-row">
  <label>True class:
    <select id="true-class">
      <option value="0">Cat</option>
      <option value="1">Fox</option>
      <option value="2">Dog</option>
    </select>
  </label>
  <label>Temperature T:
    <input id="temp" type="range" min="1" max="10" value="4" step="1">
    <span id="temp-val">4</span>
  </label>
</div>
<div id="panels">
  <div class="panel">
    <div class="panel-title teacher">Teacher output (soft)</div>
    <div id="teacher-bars" class="bars"></div>
  </div>
  <div class="panel">
    <div class="panel-title hard">Hard label (one-hot)</div>
    <div id="hard-bars" class="bars"></div>
  </div>
</div>
<div id="info-box"></div>
<div class="btns">
  <button id="btn-soft" type="button">Train student on soft targets</button>
  <button id="btn-hard" type="button">Train student on hard labels</button>
  <button id="btn-reset" type="button" class="ghost">Reset</button>
</div>
<div id="student-panels" style="display:none">
  <div class="panel">
    <div class="panel-title soft-student">Student (soft training)</div>
    <div id="soft-student-bars" class="bars"></div>
    <div class="student-info" id="soft-info"></div>
  </div>
  <div class="panel">
    <div class="panel-title hard-student">Student (hard training)</div>
    <div id="hard-student-bars" class="bars"></div>
    <div class="student-info" id="hard-info"></div>
  </div>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; font-size: 14px; }
.hint { font-size: .88rem; color: #444; margin-bottom: .8rem; line-height: 1.5; }
#scenario-row { display: flex; gap: 1.4rem; align-items: center; flex-wrap: wrap; margin-bottom: .9rem; }
#scenario-row label { display: flex; align-items: center; gap: .4rem; font-weight: 600; }
select, input[type=range] { font-size: 14px; }
#panels, #student-panels { display: grid; grid-template-columns: 1fr 1fr; gap: .8rem; margin-bottom: .8rem; }
.panel { border: 1px solid #d0d7de; border-radius: 8px; padding: .7rem .9rem; }
.panel-title { font-weight: 700; font-size: .82rem; text-transform: uppercase; letter-spacing: .04em; margin-bottom: .5rem; }
.panel-title.teacher { color: #1a7abf; }
.panel-title.hard { color: #888; }
.panel-title.soft-student { color: #0a7d33; }
.panel-title.hard-student { color: #c25700; }
.bar-row { display: flex; align-items: center; gap: .5rem; margin-bottom: .35rem; }
.bar-label { width: 34px; font-size: .82rem; font-weight: 600; flex-shrink: 0; }
.bar-track { flex: 1; height: 18px; background: #eef0f2; border-radius: 4px; overflow: hidden; }
.bar-fill { height: 100%; border-radius: 4px; transition: width .5s ease; }
.bar-fill.teacher-c { background: #4aa8e8; }
.bar-fill.hard-c { background: #bbb; }
.bar-fill.soft-student-c { background: #2db85a; }
.bar-fill.hard-student-c { background: #e07b30; }
.bar-pct { width: 38px; text-align: right; font-size: .82rem; font-variant-numeric: tabular-nums; }
#info-box { font-size: .88rem; color: #333; background: #f0f6ff; border-left: 3px solid #4aa8e8; padding: .5rem .7rem; border-radius: 4px; margin-bottom: .8rem; min-height: 2.4em; line-height: 1.5; }
.student-info { font-size: .8rem; color: #555; margin-top: .4rem; line-height: 1.4; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin-bottom: .6rem; }
button { font: 600 13px system-ui; padding: .42rem .85rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
button:hover { opacity: .88; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Knowledge Distillation demo.
// Simulates a 3-class teacher whose logits encode similarity (cat≈fox >> dog),
// then shows what a student learns from (a) soft teacher probabilities vs (b) hard one-hot labels.
// Core math: softmax with temperature, KL divergence, and a simplified gradient descent update.

const CLASSES = ['Cat', 'Fox', 'Dog'];
// Teacher's raw logits per true class (encodes that cat and fox are similar, dog is different)
const TEACHER_LOGITS = [
  [3.0, 1.2, -1.0],  // true class = Cat
  [1.0, 3.0, -0.5],  // true class = Fox
  [-0.5, 0.2, 3.5],  // true class = Dog
];

function softmax(logits, T) {
  const scaled = logits.map(x => x / T);
  const max = Math.max(...scaled);
  const exps = scaled.map(x => Math.exp(x - max));
  const sum = exps.reduce((a, b) => a + b, 0);
  return exps.map(x => x / sum);
}

function klDiv(p, q) {
  // KL(p || q), where p = target, q = student
  return p.reduce((acc, pi, i) => acc + (pi > 1e-10 ? pi * Math.log(pi / Math.max(q[i], 1e-10)) : 0), 0);
}

// Simplified student: starts from uniform, gradient descent on cross-entropy loss.
// Returns final probability distribution after 'steps' steps.
function trainStudent(target, steps, lr) {
  // Student logits start at zero (uniform)
  let logits = [0, 0, 0];
  for (let s = 0; s < steps; s++) {
    const probs = softmax(logits, 1);
    // Gradient of cross-entropy loss w.r.t. logits: grad = probs - target
    const grad = probs.map((p, i) => p - target[i]);
    logits = logits.map((l, i) => l - lr * grad[i]);
  }
  return softmax(logits, 1);
}

let trueClass = 0;
let temperature = 4;

function getTeacherProbs() {
  return softmax(TEACHER_LOGITS[trueClass], temperature);
}

function getHardLabel() {
  return CLASSES.map((_, i) => i === trueClass ? 1.0 : 0.0);
}

function renderBars(containerId, probs, colorClass) {
  const el = document.getElementById(containerId);
  el.innerHTML = '';
  probs.forEach((p, i) => {
    const row = document.createElement('div');
    row.className = 'bar-row';
    const pct = (p * 100).toFixed(1);
    row.innerHTML = `
      <span class="bar-label">${CLASSES[i]}</span>
      <div class="bar-track">
        <div class="bar-fill ${colorClass}" style="width:${pct}%"></div>
      </div>
      <span class="bar-pct">${pct}%</span>
    `;
    el.appendChild(row);
  });
}

function updateInfo() {
  const teacherProbs = getTeacherProbs();
  const hardLabel = getHardLabel();
  const box = document.getElementById('info-box');
  const secondBest = teacherProbs.map((p, i) => ({p, i}))
    .sort((a, b) => b.p - a.p)[1];
  box.innerHTML = fmt('Teacher sees <strong>%v%</strong>. With T=%temperature%, it assigns <strong>%v2%%</strong> to the true class and still gives <strong>%v3%%</strong> to <strong>%v4%</strong>. A hard label would give 100% / 0% / 0% — discarding the relational signal entirely.', { v: CLASSES[trueClass], temperature: temperature, v2: (teacherProbs[trueClass]*100).toFixed(1), v3: (secondBest.p*100).toFixed(1), v4: CLASSES[secondBest.i] });
}

function renderAll() {
  const teacherProbs = getTeacherProbs();
  const hardLabel = getHardLabel();
  renderBars('teacher-bars', teacherProbs, 'teacher-c');
  renderBars('hard-bars', hardLabel, 'hard-c');
  updateInfo();
  document.getElementById('student-panels').style.display = 'none';
}

document.getElementById('true-class').addEventListener('change', e => {
  trueClass = parseInt(e.target.value);
  renderAll();
});
document.getElementById('temp').addEventListener('input', e => {
  temperature = parseInt(e.target.value);
  document.getElementById('temp-val').textContent = temperature;
  renderAll();
});

document.getElementById('btn-soft').addEventListener('click', () => {
  const teacherProbs = getTeacherProbs();
  const hardLabel = getHardLabel();
  // Train student on soft targets
  const softStudent = trainStudent(teacherProbs, 200, 0.1);
  // Train student on hard labels
  const hardStudent = trainStudent(hardLabel, 200, 0.1);

  renderBars('soft-student-bars', softStudent, 'soft-student-c');
  renderBars('hard-student-bars', hardStudent, 'hard-student-c');

  const klSoft = klDiv(teacherProbs, softStudent);
  const klHard = klDiv(teacherProbs, hardStudent);

  document.getElementById('soft-info').innerHTML =
    fmt('KL divergence from teacher: <strong>%klSoft%</strong>. Learned the relational structure.', { klSoft: klSoft.toFixed(4) });
  document.getElementById('hard-info').innerHTML =
    fmt('KL divergence from teacher: <strong>%klHard%</strong>. Overconfident on true class.', { klHard: klHard.toFixed(4) });

  document.getElementById('student-panels').style.display = 'grid';

  console.assert(klSoft < klHard,
    fmt('Soft student should be closer to teacher than hard student: klSoft=%klSoft% klHard=%klHard%', { klSoft: klSoft, klHard: klHard }));
  console.log('Soft student KL:', klSoft.toFixed(4), '| Hard student KL:', klHard.toFixed(4),
    '| Soft better:', klSoft < klHard);
});

document.getElementById('btn-hard').addEventListener('click', () => {
  // Same as btn-soft but only trains hard student first for contrast
  document.getElementById('btn-soft').click();
});

document.getElementById('btn-reset').addEventListener('click', () => {
  document.getElementById('student-panels').style.display = 'none';
  renderAll();
});

renderAll();
Result
Console

Observe how the student trained with soft targets converges to a better-calibrated solution: it learns not just which class is correct, but how similar the wrong classes are to each other. The student trained on hard labels tends to be overconfident and misses the relationships the teacher learned. This gap is the core insight of knowledge distillation.

The Real Complexity

Knowledge distillation is solved as an engineering technique — it reliably works, and practitioners use it at scale every day. But the theoretical picture is richer than it first appears.

The temperature trick. The teacher's raw softmax outputs are often too peaked: one class gets 99.9 % and the rest nearly zero. A temperature parameter T > 1 flattens the distribution, amplifying the small probabilities that carry the most structure. At T = 1 you get the original outputs; at T = 10 the distribution spreads nearly flat. The student is trained at high temperature, then deployed at T = 1.

The loss function. The student minimizes a weighted sum of two terms:

  • Cross-entropy with hard labels — the conventional training signal.
  • KL divergence from the teacher's soft distribution — the distillation signal.

A hyperparameter α blends them. In practice, a high distillation weight (α close to 1) often dominates and produces the best results.

Why it helps — open questions. The intuitive explanation is that soft targets encode dark knowledge: relational structure the teacher learned that hard labels hide. Formally, this has been connected to ideas in PAC learning and information-theoretic compression, but a tight theory explaining exactly how much better a student can get from a given teacher remains an active research area. What is clear is that the technique transfers across domains: computer vision, natural language, speech, and even reinforcement learning use distillation routinely.

Capacity gap. Surprisingly, making the student too small relative to the teacher can hurt — the student cannot represent what the teacher knows. The optimal student size is a function of the teacher's complexity and the task, and choosing it well is still an empirical art.

Where It Matters

Knowledge distillation is one of the most widely deployed ideas in modern machine learning:

  • On-device AI: smartphone assistants, keyboard autocomplete, and real-time image processing all run distilled models small enough to fit in a few megabytes without sacrificing much accuracy.
  • Language model compression: DistilBERT (2019) distilled BERT into a model 40 % smaller and 60 % faster while retaining 97 % of its performance. The same recipe produced TinyBERT, MobileBERT, and dozens of successors.
  • Ensemble distillation: when you have a costly ensemble of many models, you can distill all their combined knowledge into a single small model — getting ensemble-level accuracy at single-model cost.
  • Privacy-preserving transfer: a teacher trained on sensitive data can teach a student on public data without exposing any individual records — the soft probabilities reveal aggregate structure, not individual examples.
  • Continual learning: distillation on old tasks while learning new ones is a leading strategy to avoid catastrophic forgetting in neural networks.

The technique also inspired a family of related methods — feature distillation, attention transfer, contrastive representation learning — all built on the same intuition: the rich internal representations of a large model are a better training signal than any human-assigned label.

Conclusion

The central insight of knowledge distillation is disarmingly simple: a wrong answer given with 7 % confidence is information, not noise. By teaching a small model to reproduce the full probability distribution of a large one — including those small, revealing confidences on the wrong classes — you transfer structural knowledge that no hard label can convey.

The result is a family of compact models that punch far above their weight: distilled language models, vision models, and speech recognizers run on devices that could never host the originals, at accuracy that would have been impossible to achieve by training small from scratch.

If you are curious about the broader landscape of what neural networks can and cannot learn, explore PAC learning for the theoretical bounds, or neural network training for the optimization story that makes all of this possible.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/knowledge-distillation/Content licensed under CC BY-NC 4.0.