CLIP

Introduction

For most of the 2010s, computer vision learned from fixed label sets. A network trained on ImageNet knows exactly 1,000 categories; ask it about a category that is not on the list and it has nothing to say. Every new task meant collecting and labelling a new dataset.

The internet, meanwhile, is full of images that already come with free text: alt text, titles, captions. In January 2021 OpenAI announced CLIP (Contrastive Language–Image Pre-training), described in the paper Learning Transferable Visual Models From Natural Language Supervision by Alec Radford, Jong Wook Kim and colleagues (arXiv, February 2021; ICML 2021). Its bet: instead of predicting a label, learn which caption goes with which image, on about 400 million image–text pairs gathered from the web (a dataset the paper calls WIT, WebImageText).

CLIP has two towers that never look at each other's input:

an image encoder — a ResNet with attention pooling or a Vision Transformer — that turns a picture into a vector;
a text encoder — a 12-layer Transformer of about 63 million parameters with a 49,152-token BPE vocabulary — that turns a caption into a vector.

Each output is linearly projected into the same space and normalised to unit length. Training then plays a matching game over a batch of $N$ pairs: of the $N \times N$ possible (image, caption) combinations, only the $N$ on the diagonal are real. The network is rewarded for making those $N$ similarities large and the other $N^2 - N$ small — the contrastive learning idea, applied across two modalities.

# one CLIP training step, numpy-style
I = normalize(image_encoder(images) @ W_img)   # N x d
T = normalize(text_encoder(texts)  @ W_txt)    # N x d
logits = exp(t) * I @ T.T                      # N x N, t is learned
labels = arange(N)                             # pair i matches caption i
loss = (cross_entropy(logits, labels, axis=1)  # image -> text
      + cross_entropy(logits, labels, axis=0)) / 2   # text -> image

That is essentially the whole algorithm. What makes it remarkable is what falls out of it for free: once images and sentences live in one space, any sentence becomes a classifier.

Train a Toy CLIP

This is a real, if tiny, CLIP. The "images" are nine coloured shapes described by seven hand-made numbers (colour and shape features); the "captions" are short phrases encoded as a bag of words. Two linear encoders project both into an 8-dimensional space, and every click runs genuine gradient steps of the symmetric contrastive loss, with a learned temperature, on a batch of eight pairs.

<p class="hint">Nine toy “photos” (coloured shapes) and their captions. Eight pairs are the training batch; the <b>green triangle</b> is held out. Two tiny linear encoders map images and captions into the same 8-D space, and the symmetric contrastive loss pulls each image toward its own caption and away from the other seven.</p>
<div class="controls">
  <button id="step1" type="button">Train 1 step</button>
  <button id="step25" type="button">Train 25 steps</button>
  <button id="auto" type="button">Auto-train</button>
  <button id="reset" type="button" class="ghost">New random init</button>
</div>
<div class="stats" id="stats"></div>
<div class="panels">
  <figure class="panel">
    <figcaption>Batch similarity matrix (rows = images, columns = captions)</figcaption>
    <canvas id="heat" width="300" height="386" aria-label="Heatmap of cosine similarities between the eight training images and the eight captions"></canvas>
    <div class="scale"><span class="sw neg"></span> −1 <span class="sw mid"></span> 0 <span class="sw pos"></span> +1 · cosine similarity; the outlined diagonal holds the true pairs</div>
  </figure>
  <figure class="panel">
    <figcaption>Contrastive loss</figcaption>
    <canvas id="loss" width="300" height="140" aria-label="Line chart of the training loss"></canvas>
  </figure>
</div>
<h4 class="zs-title">Zero-shot classification</h4>
<p class="hint">Pick an image — try the dashed one the model never saw — and write candidate captions, one per line. The model embeds each caption and turns the similarities into probabilities with a softmax, exactly like CLIP's “a photo of a {label}” trick.</p>
<div id="picker" class="picker"></div>
<label for="cands" class="lab">Candidate captions</label>
<textarea id="cands" rows="5" spellcheck="false"></textarea>
<div id="bars" class="bars"></div>
<div id="tokens" class="tokens"></div>

* { box-sizing: border-box; }
:root { --fg: #1f2933; --muted: #52606d; --line: #d9e2ec; --panel: #f7f9fb; --accent: #1d3557; --bar: #3a6fd8; }
body { font-family: system-ui, sans-serif; margin: 0; color: var(--fg); }
.hint { font-size: .86rem; color: var(--muted); margin: 0 0 .6rem; line-height: 1.45; }
.controls { display: flex; gap: .45rem; flex-wrap: wrap; margin-bottom: .5rem; }
button { font: 600 13px system-ui, sans-serif; padding: .42rem .8rem; border: 1px solid var(--accent);
         background: var(--accent); color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: transparent; color: var(--accent); }
button.on { outline: 2px solid var(--bar); outline-offset: 2px; }
.stats { font: 600 13px ui-monospace, monospace; color: var(--fg); margin: .2rem 0 .6rem; min-height: 1.3em; }
.panels { display: flex; flex-wrap: wrap; gap: .8rem; }
.panel { margin: 0; padding: .5rem; background: var(--panel); border: 1px solid var(--line); border-radius: 10px; }
figcaption { font-size: .82rem; font-weight: 600; color: var(--muted); margin-bottom: .3rem; }
canvas { display: block; width: 100%; max-width: 300px; height: auto; }
.scale { font-size: .75rem; color: var(--muted); margin-top: .3rem; display: flex; align-items: center; gap: .25rem; flex-wrap: wrap; }
.sw { display: inline-block; width: 12px; height: 12px; border-radius: 3px; }
.sw.neg { background: #e76f51; } .sw.mid { background: #e9edf2; border: 1px solid var(--line); } .sw.pos { background: #2b6cd4; }
.zs-title { margin: 1rem 0 .3rem; font-size: 1rem; }
.picker { display: flex; flex-wrap: wrap; gap: .35rem; margin-bottom: .5rem; }
.pick { position: relative; padding: 2px; border: 2px solid var(--line); border-radius: 8px; background: var(--panel); cursor: pointer; }
.pick.sel { border-color: var(--bar); }
.pick.unseen { border-style: dashed; }
.pick canvas { width: 36px; height: 36px; }
.badge { position: absolute; top: -8px; right: -6px; font: 700 9px system-ui; background: #e9c46a; color: #222; border-radius: 6px; padding: 1px 4px; }
.lab { display: block; font-size: .82rem; font-weight: 600; color: var(--muted); margin: .3rem 0 .2rem; }
textarea { width: 100%; max-width: 420px; font: 13px system-ui, sans-serif; padding: .4rem; border-radius: 8px;
           border: 1px solid var(--line); background: var(--panel); color: var(--fg); }
.bars { margin-top: .5rem; max-width: 420px; }
.row { display: grid; grid-template-columns: minmax(0, 1fr) 90px 44px; gap: .4rem; align-items: center; font-size: .82rem; margin: .18rem 0; }
.row .t { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.row.best .t { font-weight: 700; }
.track { height: 10px; background: var(--line); border-radius: 5px; overflow: hidden; }
.fill { height: 100%; background: var(--bar); }
.row .v { font: 12px ui-monospace, monospace; text-align: right; }
.tokens { font-size: .78rem; color: var(--muted); margin-top: .35rem; }
@media (prefers-color-scheme: dark) {
  :root { --fg: #e4e9ef; --muted: #a3b1bf; --line: #34404d; --panel: #1b222b; --accent: #9fb2c8; --bar: #6ea0ff; }
  body { background: #12171d; }
  button { color: #12171d; }
  button.ghost { background: transparent; color: var(--accent); }
  .sw.mid { background: #28303a; }
}

const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Toy CLIP: an image encoder and a text encoder (both linear maps to 8-D, then L2-normalised)
// are trained together with the symmetric InfoNCE loss over the N×N similarity matrix of a batch.
// The temperature is learned too (as log of the logit scale), initialised at 0.07 and clipped at 1/100, as in the paper.
const COLORS = [
  { key: 'red', word: "red", rgb: [0.95, 0.2, 0.2], css: '#e63946' },
  { key: 'green', word: "green", rgb: [0.2, 0.8, 0.3], css: '#2a9d4f' },
  { key: 'blue', word: "blue", rgb: [0.2, 0.35, 0.95], css: '#3a6fd8' }
];
// Hand-made image features: RGB colour plus three shape descriptors (corners, roundness, pointiness) and a bias 1. Real CLIP sees raw pixels.
const SHAPES = [
  { key: 'circle', word: "circle", feat: [0, 1, 0] },
  { key: 'square', word: "square", feat: [1, 0, 0] },
  { key: 'triangle', word: "triangle", feat: [0.6, 0, 1] }
];
const TEMPLATE = "a %color% %shape%";
const caption = (c, s) => i18nFmt(TEMPLATE, { color: c.word, shape: s.word });

// Bag-of-words text features: a vocabulary word counts if it appears in the caption (case- and accent-insensitive).
const norm = (t) => t.toLowerCase().normalize('NFD').replace(/[̀-ͯ]/g, '');
const VOCAB = [...COLORS.map(c => c.word), ...SHAPES.map(s => s.word)];
const VOCAB_N = VOCAB.map(norm);
function textFeat(str) {
  const t = norm(str);
  const f = VOCAB_N.map(w => (t.includes(w) ? 1 : 0));
  f.push(1); // constant feature, plays the role of filler words such as “a photo of”
  return f;
}

// 3 colours × 3 shapes = 9 image–caption pairs; the green triangle is held out for zero-shot testing.
const ALL = [];
for (const c of COLORS) for (const s of SHAPES) {
  ALL.push({ c, s, img: [...c.rgb, ...s.feat, 1], cap: caption(c, s),
             unseen: c.key === 'green' && s.key === 'triangle' });
}
ALL.forEach(p => { p.txt = textFeat(p.cap); });
const TRAIN = ALL.filter(p => !p.unseen);
const N = TRAIN.length, DI = 7, DT = 7, D = 8;
const MAX_LOGS = Math.log(100); // the logit scale may not exceed 100 (temperature ≥ 0.01)

let Wi, Wt, logS, adam, stepCount, lossHist, autoTimer = null, lastF = null;
function randn() { let u = 0, v = 0; while (!u) u = Math.random(); while (!v) v = Math.random();
  return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v); }
const mat = (r, c, sd) => Array.from({ length: r }, () => Array.from({ length: c }, () => randn() * sd));
const zeros = (r, c) => Array.from({ length: r }, () => new Array(c).fill(0));
const dot = (a, b) => a.reduce((s, x, i) => s + x * b[i], 0);

function encode(W, x) {
  const a = W.map(row => dot(row, x));
  const n = Math.sqrt(dot(a, a)) + 1e-9;
  return { a, n, u: a.map(v => v / n) };
}

// Forward pass: embed, normalise, cosine matrix × scale, softmax over rows (image→text) and over columns (text→image).
function forward() {
  const I = TRAIN.map(p => encode(Wi, p.img));
  const T = TRAIN.map(p => encode(Wt, p.txt));
  const s = Math.exp(logS);
  const cos = I.map(ii => T.map(tt => dot(ii.u, tt.u)));
  const L = cos.map(r => r.map(v => s * v));
  const rowP = L.map(r => { const m = Math.max(...r); const e = r.map(v => Math.exp(v - m)); const z = e.reduce((a, b) => a + b, 0); return e.map(v => v / z); });
  const colP = zeros(N, N);
  for (let j = 0; j < N; j++) {
    let m = -Infinity; for (let i = 0; i < N; i++) m = Math.max(m, L[i][j]);
    let z = 0; for (let i = 0; i < N; i++) z += Math.exp(L[i][j] - m);
    for (let i = 0; i < N; i++) colP[i][j] = Math.exp(L[i][j] - m) / z;
  }
  let loss = 0;
  for (let i = 0; i < N; i++) loss -= Math.log(rowP[i][i] + 1e-12) + Math.log(colP[i][i] + 1e-12);
  loss /= 2 * N;
  return { I, T, s, cos, L, rowP, colP, loss };
}

// Backward pass: dLoss/dlogits = (P_rows + P_cols − 2·I) / 2N, then chain rule through the scale, the normalisation and the linear maps.
function gradients(F) {
  const G = zeros(N, N);
  for (let i = 0; i < N; i++) for (let j = 0; j < N; j++)
    G[i][j] = (F.rowP[i][j] + F.colP[i][j] - 2 * (i === j ? 1 : 0)) / (2 * N);
  const gWi = zeros(D, DI), gWt = zeros(D, DT);
  let gLogS = 0;
  for (let i = 0; i < N; i++) for (let j = 0; j < N; j++) gLogS += G[i][j] * F.L[i][j];
  const back = (enc, gu, x, gW) => {
    const proj = dot(gu, enc.u);
    for (let r = 0; r < gu.length; r++) {
      const ga = (gu[r] - proj * enc.u[r]) / enc.n;
      for (let c = 0; c < x.length; c++) gW[r][c] += ga * x[c];
    }
  };
  for (let i = 0; i < N; i++) {
    const gu = new Array(D).fill(0);
    for (let j = 0; j < N; j++) for (let k = 0; k < D; k++) gu[k] += F.s * G[i][j] * F.T[j].u[k];
    back(F.I[i], gu, TRAIN[i].img, gWi);
  }
  for (let j = 0; j < N; j++) {
    const gv = new Array(D).fill(0);
    for (let i = 0; i < N; i++) for (let k = 0; k < D; k++) gv[k] += F.s * G[i][j] * F.I[i].u[k];
    back(F.T[j], gv, TRAIN[j].txt, gWt);
  }
  return { gWi, gWt, gLogS };
}

// Plain Adam optimiser on both weight matrices and on the log-scale.
const LR = 0.03, B1 = 0.9, B2 = 0.999;
function adamStep(key, W, g, t) {
  const st = adam[key];
  for (let r = 0; r < W.length; r++) for (let c = 0; c < W[r].length; c++) {
    st.m[r][c] = B1 * st.m[r][c] + (1 - B1) * g[r][c];
    st.v[r][c] = B2 * st.v[r][c] + (1 - B2) * g[r][c] * g[r][c];
    const mh = st.m[r][c] / (1 - Math.pow(B1, t)), vh = st.v[r][c] / (1 - Math.pow(B2, t));
    W[r][c] -= LR * mh / (Math.sqrt(vh) + 1e-8);
  }
}
function trainStep() {
  const F = forward();
  const g = gradients(F);
  stepCount++;
  adamStep('i', Wi, g.gWi, stepCount);
  adamStep('t', Wt, g.gWt, stepCount);
  const W = [[logS]];
  adamStep('s', W, [[g.gLogS]], stepCount);
  logS = Math.min(MAX_LOGS, W[0][0]);
  lossHist.push(F.loss);
}

function reset() {
  Wi = mat(D, DI, 0.6); Wt = mat(D, DT, 0.6);
  logS = Math.log(1 / 0.07); // temperature τ = 0.07, i.e. logit scale ≈ 14.3
  adam = { i: { m: zeros(D, DI), v: zeros(D, DI) }, t: { m: zeros(D, DT), v: zeros(D, DT) }, s: { m: [[0]], v: [[0]] } };
  stepCount = 0; lossHist = [];
  refresh();
}

// Zero-shot: embed the image and every candidate caption, softmax over scaled cosines.
function zeroShot(img, captions) {
  const u = encode(Wi, img).u;
  const s = Math.exp(logS);
  const sc = captions.map(c => s * dot(u, encode(Wt, textFeat(c)).u));
  const m = Math.max(...sc);
  const e = sc.map(v => Math.exp(v - m));
  const z = e.reduce((a, b) => a + b, 0);
  return e.map(v => v / z);
}

const dark = !!(window.matchMedia && window.matchMedia('(prefers-color-scheme: dark)').matches);
const THEME = dark ? { fg: '#e4e9ef', muted: '#a3b1bf', line: '#34404d', mid: [40, 48, 58], diag: '#e9c46a' }
                   : { fg: '#1f2933', muted: '#52606d', line: '#d9e2ec', mid: [233, 237, 242], diag: '#111111' };
function heatColor(v) {
  const t = Math.max(-1, Math.min(1, v));
  const to = t >= 0 ? [43, 108, 212] : [231, 111, 81];
  const a = Math.abs(t);
  const c = THEME.mid.map((m, k) => Math.round(m + (to[k] - m) * a));
  return 'rgb(' + c.join(',') + ')';
}
function drawShape(ctx, shape, color, cx, cy, r) {
  ctx.beginPath(); ctx.fillStyle = color;
  if (shape === 'circle') ctx.arc(cx, cy, r, 0, Math.PI * 2);
  else if (shape === 'square') ctx.rect(cx - r * 0.85, cy - r * 0.85, r * 1.7, r * 1.7);
  else { ctx.moveTo(cx, cy - r); ctx.lineTo(cx + r * 0.95, cy + r * 0.75); ctx.lineTo(cx - r * 0.95, cy + r * 0.75); ctx.closePath(); }
  ctx.fill();
}

function drawHeat(F) {
  const cv = document.getElementById('heat'), ctx = cv.getContext('2d');
  const left = 34, top = 120, cell = (cv.width - left - 4) / N;
  ctx.clearRect(0, 0, cv.width, cv.height);
  // column labels are the captions, written vertically
  for (let j = 0; j < N; j++) {
    ctx.save();
    ctx.translate(left + (j + 0.5) * cell + 4, top - 6);
    ctx.rotate(-Math.PI / 2);
    let fs = 11; ctx.font = fs + 'px system-ui, sans-serif';
    while (ctx.measureText(TRAIN[j].cap).width > top - 10 && fs > 7) { fs -= 0.5; ctx.font = fs + 'px system-ui, sans-serif'; }
    ctx.fillStyle = THEME.fg; ctx.textAlign = 'left';
    ctx.fillText(TRAIN[j].cap, 0, 0);
    ctx.restore();
  }
  for (let i = 0; i < N; i++) {
    drawShape(ctx, TRAIN[i].s.key, TRAIN[i].c.css, left / 2, top + (i + 0.5) * cell, cell * 0.32);
    for (let j = 0; j < N; j++) {
      const x = left + j * cell, y = top + i * cell, v = F.cos[i][j];
      ctx.fillStyle = heatColor(v);
      ctx.fillRect(x + 1, y + 1, cell - 2, cell - 2);
      if (i === j) { ctx.strokeStyle = THEME.diag; ctx.lineWidth = 1.5; ctx.strokeRect(x + 1.5, y + 1.5, cell - 3, cell - 3); }
      ctx.fillStyle = Math.abs(v) > 0.55 ? '#ffffff' : THEME.fg;
      ctx.font = '10px ui-monospace, monospace'; ctx.textAlign = 'center'; ctx.textBaseline = 'middle';
      ctx.fillText(v.toFixed(1), x + cell / 2, y + cell / 2);
    }
  }
  ctx.textBaseline = 'alphabetic';
}

function drawLoss() {
  const cv = document.getElementById('loss'), ctx = cv.getContext('2d');
  const W = cv.width, H = cv.height, p = 26;
  ctx.clearRect(0, 0, W, H);
  ctx.strokeStyle = THEME.line; ctx.lineWidth = 1;
  ctx.beginPath(); ctx.moveTo(p, 6); ctx.lineTo(p, H - p); ctx.lineTo(W - 6, H - p); ctx.stroke();
  const ymax = Math.max(Math.log(N) * 1.2, ...lossHist.slice(0, 5));
  ctx.fillStyle = THEME.muted; ctx.font = '10px ui-monospace, monospace'; ctx.textAlign = 'right';
  ctx.fillText(ymax.toFixed(1), p - 3, 14); ctx.fillText('0', p - 3, H - p);
  ctx.textAlign = 'left';
  ctx.fillText('ln ' + N + ' = ' + Math.log(N).toFixed(2), p + 4, H - p - 4 - (H - p - 10) * Math.log(N) / ymax);
  ctx.setLineDash([3, 3]); ctx.beginPath();
  const yl = H - p - (H - p - 10) * Math.log(N) / ymax; ctx.moveTo(p, yl); ctx.lineTo(W - 6, yl); ctx.stroke(); ctx.setLineDash([]);
  ctx.textAlign = 'center'; ctx.fillText("training steps", (W + p) / 2, H - 8);
  if (lossHist.length < 2) return;
  const xmax = Math.max(100, lossHist.length);
  ctx.strokeStyle = '#2b6cd4'; ctx.lineWidth = 2; ctx.beginPath();
  lossHist.forEach((l, k) => {
    const x = p + (W - 6 - p) * k / xmax, y = H - p - (H - p - 10) * Math.min(l, ymax) / ymax;
    if (k) ctx.lineTo(x, y); else ctx.moveTo(x, y);
  });
  ctx.stroke();
}

let selected = ALL.findIndex(p => p.unseen);
function buildPicker() {
  const box = document.getElementById('picker'); box.innerHTML = '';
  ALL.forEach((p, k) => {
    const b = document.createElement('button');
    b.type = 'button'; b.className = 'pick' + (k === selected ? ' sel' : '') + (p.unseen ? ' unseen' : '');
    b.title = p.unseen ? "Held-out image, never seen during training" : p.cap;
    b.setAttribute('aria-label', p.unseen ? "Held-out image, never seen during training" : p.cap);
    const cv = document.createElement('canvas'); cv.width = 36; cv.height = 36;
    drawShape(cv.getContext('2d'), p.s.key, p.c.css, 18, 18, 13);
    b.appendChild(cv);
    if (p.unseen) { const t = document.createElement('span'); t.className = 'badge'; t.textContent = "new"; b.appendChild(t); }
    b.addEventListener('click', () => { selected = k; buildPicker(); renderZeroShot(); });
    box.appendChild(b);
  });
}

function renderZeroShot() {
  const caps = document.getElementById('cands').value.split(/[,，、;\n]+/).map(t => t.trim()).filter(Boolean).slice(0, 12);
  const bars = document.getElementById('bars'); bars.innerHTML = '';
  if (!caps.length) return;
  const pr = zeroShot(ALL[selected].img, caps);
  const best = pr.indexOf(Math.max(...pr));
  caps.forEach((c, k) => {
    const row = document.createElement('div'); row.className = 'row' + (k === best ? ' best' : '');
    const t = document.createElement('span'); t.className = 't'; t.textContent = (k === best ? '▶ ' : '') + c; t.title = c;
    const tr = document.createElement('div'); tr.className = 'track';
    const f = document.createElement('div'); f.className = 'fill'; f.style.width = (100 * pr[k]).toFixed(1) + '%'; tr.appendChild(f);
    const v = document.createElement('span'); v.className = 'v'; v.textContent = (100 * pr[k]).toFixed(0) + '%';
    row.append(t, tr, v); bars.appendChild(row);
  });
  const seen = VOCAB.filter((w, k) => caps.some(c => norm(c).includes(VOCAB_N[k])));
  document.getElementById('tokens').textContent = i18nFmt("Words the text encoder recognised: %words%. Its whole vocabulary: %vocab%.", { words: seen.join(' · ') || '—', vocab: VOCAB.join(' · ') });
}

function refresh() {
  const F = forward(); lastF = F;
  let acc = 0;
  for (let i = 0; i < N; i++) if (F.rowP[i].indexOf(Math.max(...F.rowP[i])) === i) acc++;
  document.getElementById('stats').textContent = i18nFmt("step %step% · loss %loss% · τ = %tau% · correct matches %acc%/%n%", {
    step: stepCount, loss: F.loss.toFixed(3), tau: (1 / F.s).toFixed(3), acc: acc, n: N });
  drawHeat(F); drawLoss(); renderZeroShot();
}

function train(k) { for (let r = 0; r < k; r++) trainStep(); refresh(); }
function stopAuto() { if (autoTimer) { clearInterval(autoTimer); autoTimer = null; } document.getElementById('auto').classList.remove('on'); document.getElementById('auto').textContent = "Auto-train"; }
document.getElementById('step1').addEventListener('click', () => train(1));
document.getElementById('step25').addEventListener('click', () => train(25));
document.getElementById('auto').addEventListener('click', () => {
  if (autoTimer) { stopAuto(); return; }
  const b = document.getElementById('auto'); b.classList.add('on'); b.textContent = "Pause";
  autoTimer = setInterval(() => { train(3); if (stepCount >= 600) stopAuto(); }, 60);
});
document.getElementById('reset').addEventListener('click', () => { stopAuto(); reset(); });
document.getElementById('cands').addEventListener('input', renderZeroShot);

// Default candidates: every shape in green, plus two triangles of other colours.
const green = COLORS[1];
document.getElementById('cands').value = [
  ...SHAPES.map(s => caption(green, s)), caption(COLORS[0], SHAPES[2]), caption(COLORS[2], SHAPES[2])
].join('\n');
buildPicker();
reset();

<p class="hint">Nine toy “photos” (coloured shapes) and their captions. Eight pairs are the training batch; the <b>green triangle</b> is held out. Two tiny linear encoders map images and captions into the same 8-D space, and the symmetric contrastive loss pulls each image toward its own caption and away from the other seven.</p>
<div class="controls">
  <button id="step1" type="button">Train 1 step</button>
  <button id="step25" type="button">Train 25 steps</button>
  <button id="auto" type="button">Auto-train</button>
  <button id="reset" type="button" class="ghost">New random init</button>
</div>
<div class="stats" id="stats"></div>
<div class="panels">
  <figure class="panel">
    <figcaption>Batch similarity matrix (rows = images, columns = captions)</figcaption>
    <canvas id="heat" width="300" height="386" aria-label="Heatmap of cosine similarities between the eight training images and the eight captions"></canvas>
    <div class="scale"><span class="sw neg"></span> −1 <span class="sw mid"></span> 0 <span class="sw pos"></span> +1 · cosine similarity; the outlined diagonal holds the true pairs</div>
  </figure>
  <figure class="panel">
    <figcaption>Contrastive loss</figcaption>
    <canvas id="loss" width="300" height="140" aria-label="Line chart of the training loss"></canvas>
  </figure>
</div>
<h4 class="zs-title">Zero-shot classification</h4>
<p class="hint">Pick an image — try the dashed one the model never saw — and write candidate captions, one per line. The model embeds each caption and turns the similarities into probabilities with a softmax, exactly like CLIP's “a photo of a {label}” trick.</p>
<div id="picker" class="picker"></div>
<label for="cands" class="lab">Candidate captions</label>
<textarea id="cands" rows="5" spellcheck="false"></textarea>
<div id="bars" class="bars"></div>
<div id="tokens" class="tokens"></div>

* { box-sizing: border-box; }
:root { --fg: #1f2933; --muted: #52606d; --line: #d9e2ec; --panel: #f7f9fb; --accent: #1d3557; --bar: #3a6fd8; }
body { font-family: system-ui, sans-serif; margin: 0; color: var(--fg); }
.hint { font-size: .86rem; color: var(--muted); margin: 0 0 .6rem; line-height: 1.45; }
.controls { display: flex; gap: .45rem; flex-wrap: wrap; margin-bottom: .5rem; }
button { font: 600 13px system-ui, sans-serif; padding: .42rem .8rem; border: 1px solid var(--accent);
         background: var(--accent); color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: transparent; color: var(--accent); }
button.on { outline: 2px solid var(--bar); outline-offset: 2px; }
.stats { font: 600 13px ui-monospace, monospace; color: var(--fg); margin: .2rem 0 .6rem; min-height: 1.3em; }
.panels { display: flex; flex-wrap: wrap; gap: .8rem; }
.panel { margin: 0; padding: .5rem; background: var(--panel); border: 1px solid var(--line); border-radius: 10px; }
figcaption { font-size: .82rem; font-weight: 600; color: var(--muted); margin-bottom: .3rem; }
canvas { display: block; width: 100%; max-width: 300px; height: auto; }
.scale { font-size: .75rem; color: var(--muted); margin-top: .3rem; display: flex; align-items: center; gap: .25rem; flex-wrap: wrap; }
.sw { display: inline-block; width: 12px; height: 12px; border-radius: 3px; }
.sw.neg { background: #e76f51; } .sw.mid { background: #e9edf2; border: 1px solid var(--line); } .sw.pos { background: #2b6cd4; }
.zs-title { margin: 1rem 0 .3rem; font-size: 1rem; }
.picker { display: flex; flex-wrap: wrap; gap: .35rem; margin-bottom: .5rem; }
.pick { position: relative; padding: 2px; border: 2px solid var(--line); border-radius: 8px; background: var(--panel); cursor: pointer; }
.pick.sel { border-color: var(--bar); }
.pick.unseen { border-style: dashed; }
.pick canvas { width: 36px; height: 36px; }
.badge { position: absolute; top: -8px; right: -6px; font: 700 9px system-ui; background: #e9c46a; color: #222; border-radius: 6px; padding: 1px 4px; }
.lab { display: block; font-size: .82rem; font-weight: 600; color: var(--muted); margin: .3rem 0 .2rem; }
textarea { width: 100%; max-width: 420px; font: 13px system-ui, sans-serif; padding: .4rem; border-radius: 8px;
           border: 1px solid var(--line); background: var(--panel); color: var(--fg); }
.bars { margin-top: .5rem; max-width: 420px; }
.row { display: grid; grid-template-columns: minmax(0, 1fr) 90px 44px; gap: .4rem; align-items: center; font-size: .82rem; margin: .18rem 0; }
.row .t { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.row.best .t { font-weight: 700; }
.track { height: 10px; background: var(--line); border-radius: 5px; overflow: hidden; }
.fill { height: 100%; background: var(--bar); }
.row .v { font: 12px ui-monospace, monospace; text-align: right; }
.tokens { font-size: .78rem; color: var(--muted); margin-top: .35rem; }
@media (prefers-color-scheme: dark) {
  :root { --fg: #e4e9ef; --muted: #a3b1bf; --line: #34404d; --panel: #1b222b; --accent: #9fb2c8; --bar: #6ea0ff; }
  body { background: #12171d; }
  button { color: #12171d; }
  button.ghost { background: transparent; color: var(--accent); }
  .sw.mid { background: #28303a; }
}

const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Toy CLIP: an image encoder and a text encoder (both linear maps to 8-D, then L2-normalised)
// are trained together with the symmetric InfoNCE loss over the N×N similarity matrix of a batch.
// The temperature is learned too (as log of the logit scale), initialised at 0.07 and clipped at 1/100, as in the paper.
const COLORS = [
  { key: 'red', word: "red", rgb: [0.95, 0.2, 0.2], css: '#e63946' },
  { key: 'green', word: "green", rgb: [0.2, 0.8, 0.3], css: '#2a9d4f' },
  { key: 'blue', word: "blue", rgb: [0.2, 0.35, 0.95], css: '#3a6fd8' }
];
// Hand-made image features: RGB colour plus three shape descriptors (corners, roundness, pointiness) and a bias 1. Real CLIP sees raw pixels.
const SHAPES = [
  { key: 'circle', word: "circle", feat: [0, 1, 0] },
  { key: 'square', word: "square", feat: [1, 0, 0] },
  { key: 'triangle', word: "triangle", feat: [0.6, 0, 1] }
];
const TEMPLATE = "a %color% %shape%";
const caption = (c, s) => i18nFmt(TEMPLATE, { color: c.word, shape: s.word });

// Bag-of-words text features: a vocabulary word counts if it appears in the caption (case- and accent-insensitive).
const norm = (t) => t.toLowerCase().normalize('NFD').replace(/[̀-ͯ]/g, '');
const VOCAB = [...COLORS.map(c => c.word), ...SHAPES.map(s => s.word)];
const VOCAB_N = VOCAB.map(norm);
function textFeat(str) {
  const t = norm(str);
  const f = VOCAB_N.map(w => (t.includes(w) ? 1 : 0));
  f.push(1); // constant feature, plays the role of filler words such as “a photo of”
  return f;
}

// 3 colours × 3 shapes = 9 image–caption pairs; the green triangle is held out for zero-shot testing.
const ALL = [];
for (const c of COLORS) for (const s of SHAPES) {
  ALL.push({ c, s, img: [...c.rgb, ...s.feat, 1], cap: caption(c, s),
             unseen: c.key === 'green' && s.key === 'triangle' });
}
ALL.forEach(p => { p.txt = textFeat(p.cap); });
const TRAIN = ALL.filter(p => !p.unseen);
const N = TRAIN.length, DI = 7, DT = 7, D = 8;
const MAX_LOGS = Math.log(100); // the logit scale may not exceed 100 (temperature ≥ 0.01)

let Wi, Wt, logS, adam, stepCount, lossHist, autoTimer = null, lastF = null;
function randn() { let u = 0, v = 0; while (!u) u = Math.random(); while (!v) v = Math.random();
  return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v); }
const mat = (r, c, sd) => Array.from({ length: r }, () => Array.from({ length: c }, () => randn() * sd));
const zeros = (r, c) => Array.from({ length: r }, () => new Array(c).fill(0));
const dot = (a, b) => a.reduce((s, x, i) => s + x * b[i], 0);

function encode(W, x) {
  const a = W.map(row => dot(row, x));
  const n = Math.sqrt(dot(a, a)) + 1e-9;
  return { a, n, u: a.map(v => v / n) };
}

// Forward pass: embed, normalise, cosine matrix × scale, softmax over rows (image→text) and over columns (text→image).
function forward() {
  const I = TRAIN.map(p => encode(Wi, p.img));
  const T = TRAIN.map(p => encode(Wt, p.txt));
  const s = Math.exp(logS);
  const cos = I.map(ii => T.map(tt => dot(ii.u, tt.u)));
  const L = cos.map(r => r.map(v => s * v));
  const rowP = L.map(r => { const m = Math.max(...r); const e = r.map(v => Math.exp(v - m)); const z = e.reduce((a, b) => a + b, 0); return e.map(v => v / z); });
  const colP = zeros(N, N);
  for (let j = 0; j < N; j++) {
    let m = -Infinity; for (let i = 0; i < N; i++) m = Math.max(m, L[i][j]);
    let z = 0; for (let i = 0; i < N; i++) z += Math.exp(L[i][j] - m);
    for (let i = 0; i < N; i++) colP[i][j] = Math.exp(L[i][j] - m) / z;
  }
  let loss = 0;
  for (let i = 0; i < N; i++) loss -= Math.log(rowP[i][i] + 1e-12) + Math.log(colP[i][i] + 1e-12);
  loss /= 2 * N;
  return { I, T, s, cos, L, rowP, colP, loss };
}

// Backward pass: dLoss/dlogits = (P_rows + P_cols − 2·I) / 2N, then chain rule through the scale, the normalisation and the linear maps.
function gradients(F) {
  const G = zeros(N, N);
  for (let i = 0; i < N; i++) for (let j = 0; j < N; j++)
    G[i][j] = (F.rowP[i][j] + F.colP[i][j] - 2 * (i === j ? 1 : 0)) / (2 * N);
  const gWi = zeros(D, DI), gWt = zeros(D, DT);
  let gLogS = 0;
  for (let i = 0; i < N; i++) for (let j = 0; j < N; j++) gLogS += G[i][j] * F.L[i][j];
  const back = (enc, gu, x, gW) => {
    const proj = dot(gu, enc.u);
    for (let r = 0; r < gu.length; r++) {
      const ga = (gu[r] - proj * enc.u[r]) / enc.n;
      for (let c = 0; c < x.length; c++) gW[r][c] += ga * x[c];
    }
  };
  for (let i = 0; i < N; i++) {
    const gu = new Array(D).fill(0);
    for (let j = 0; j < N; j++) for (let k = 0; k < D; k++) gu[k] += F.s * G[i][j] * F.T[j].u[k];
    back(F.I[i], gu, TRAIN[i].img, gWi);
  }
  for (let j = 0; j < N; j++) {
    const gv = new Array(D).fill(0);
    for (let i = 0; i < N; i++) for (let k = 0; k < D; k++) gv[k] += F.s * G[i][j] * F.I[i].u[k];
    back(F.T[j], gv, TRAIN[j].txt, gWt);
  }
  return { gWi, gWt, gLogS };
}

// Plain Adam optimiser on both weight matrices and on the log-scale.
const LR = 0.03, B1 = 0.9, B2 = 0.999;
function adamStep(key, W, g, t) {
  const st = adam[key];
  for (let r = 0; r < W.length; r++) for (let c = 0; c < W[r].length; c++) {
    st.m[r][c] = B1 * st.m[r][c] + (1 - B1) * g[r][c];
    st.v[r][c] = B2 * st.v[r][c] + (1 - B2) * g[r][c] * g[r][c];
    const mh = st.m[r][c] / (1 - Math.pow(B1, t)), vh = st.v[r][c] / (1 - Math.pow(B2, t));
    W[r][c] -= LR * mh / (Math.sqrt(vh) + 1e-8);
  }
}
function trainStep() {
  const F = forward();
  const g = gradients(F);
  stepCount++;
  adamStep('i', Wi, g.gWi, stepCount);
  adamStep('t', Wt, g.gWt, stepCount);
  const W = [[logS]];
  adamStep('s', W, [[g.gLogS]], stepCount);
  logS = Math.min(MAX_LOGS, W[0][0]);
  lossHist.push(F.loss);
}

function reset() {
  Wi = mat(D, DI, 0.6); Wt = mat(D, DT, 0.6);
  logS = Math.log(1 / 0.07); // temperature τ = 0.07, i.e. logit scale ≈ 14.3
  adam = { i: { m: zeros(D, DI), v: zeros(D, DI) }, t: { m: zeros(D, DT), v: zeros(D, DT) }, s: { m: [[0]], v: [[0]] } };
  stepCount = 0; lossHist = [];
  refresh();
}

// Zero-shot: embed the image and every candidate caption, softmax over scaled cosines.
function zeroShot(img, captions) {
  const u = encode(Wi, img).u;
  const s = Math.exp(logS);
  const sc = captions.map(c => s * dot(u, encode(Wt, textFeat(c)).u));
  const m = Math.max(...sc);
  const e = sc.map(v => Math.exp(v - m));
  const z = e.reduce((a, b) => a + b, 0);
  return e.map(v => v / z);
}

const dark = !!(window.matchMedia && window.matchMedia('(prefers-color-scheme: dark)').matches);
const THEME = dark ? { fg: '#e4e9ef', muted: '#a3b1bf', line: '#34404d', mid: [40, 48, 58], diag: '#e9c46a' }
                   : { fg: '#1f2933', muted: '#52606d', line: '#d9e2ec', mid: [233, 237, 242], diag: '#111111' };
function heatColor(v) {
  const t = Math.max(-1, Math.min(1, v));
  const to = t >= 0 ? [43, 108, 212] : [231, 111, 81];
  const a = Math.abs(t);
  const c = THEME.mid.map((m, k) => Math.round(m + (to[k] - m) * a));
  return 'rgb(' + c.join(',') + ')';
}
function drawShape(ctx, shape, color, cx, cy, r) {
  ctx.beginPath(); ctx.fillStyle = color;
  if (shape === 'circle') ctx.arc(cx, cy, r, 0, Math.PI * 2);
  else if (shape === 'square') ctx.rect(cx - r * 0.85, cy - r * 0.85, r * 1.7, r * 1.7);
  else { ctx.moveTo(cx, cy - r); ctx.lineTo(cx + r * 0.95, cy + r * 0.75); ctx.lineTo(cx - r * 0.95, cy + r * 0.75); ctx.closePath(); }
  ctx.fill();
}

function drawHeat(F) {
  const cv = document.getElementById('heat'), ctx = cv.getContext('2d');
  const left = 34, top = 120, cell = (cv.width - left - 4) / N;
  ctx.clearRect(0, 0, cv.width, cv.height);
  // column labels are the captions, written vertically
  for (let j = 0; j < N; j++) {
    ctx.save();
    ctx.translate(left + (j + 0.5) * cell + 4, top - 6);
    ctx.rotate(-Math.PI / 2);
    let fs = 11; ctx.font = fs + 'px system-ui, sans-serif';
    while (ctx.measureText(TRAIN[j].cap).width > top - 10 && fs > 7) { fs -= 0.5; ctx.font = fs + 'px system-ui, sans-serif'; }
    ctx.fillStyle = THEME.fg; ctx.textAlign = 'left';
    ctx.fillText(TRAIN[j].cap, 0, 0);
    ctx.restore();
  }
  for (let i = 0; i < N; i++) {
    drawShape(ctx, TRAIN[i].s.key, TRAIN[i].c.css, left / 2, top + (i + 0.5) * cell, cell * 0.32);
    for (let j = 0; j < N; j++) {
      const x = left + j * cell, y = top + i * cell, v = F.cos[i][j];
      ctx.fillStyle = heatColor(v);
      ctx.fillRect(x + 1, y + 1, cell - 2, cell - 2);
      if (i === j) { ctx.strokeStyle = THEME.diag; ctx.lineWidth = 1.5; ctx.strokeRect(x + 1.5, y + 1.5, cell - 3, cell - 3); }
      ctx.fillStyle = Math.abs(v) > 0.55 ? '#ffffff' : THEME.fg;
      ctx.font = '10px ui-monospace, monospace'; ctx.textAlign = 'center'; ctx.textBaseline = 'middle';
      ctx.fillText(v.toFixed(1), x + cell / 2, y + cell / 2);
    }
  }
  ctx.textBaseline = 'alphabetic';
}

function drawLoss() {
  const cv = document.getElementById('loss'), ctx = cv.getContext('2d');
  const W = cv.width, H = cv.height, p = 26;
  ctx.clearRect(0, 0, W, H);
  ctx.strokeStyle = THEME.line; ctx.lineWidth = 1;
  ctx.beginPath(); ctx.moveTo(p, 6); ctx.lineTo(p, H - p); ctx.lineTo(W - 6, H - p); ctx.stroke();
  const ymax = Math.max(Math.log(N) * 1.2, ...lossHist.slice(0, 5));
  ctx.fillStyle = THEME.muted; ctx.font = '10px ui-monospace, monospace'; ctx.textAlign = 'right';
  ctx.fillText(ymax.toFixed(1), p - 3, 14); ctx.fillText('0', p - 3, H - p);
  ctx.textAlign = 'left';
  ctx.fillText('ln ' + N + ' = ' + Math.log(N).toFixed(2), p + 4, H - p - 4 - (H - p - 10) * Math.log(N) / ymax);
  ctx.setLineDash([3, 3]); ctx.beginPath();
  const yl = H - p - (H - p - 10) * Math.log(N) / ymax; ctx.moveTo(p, yl); ctx.lineTo(W - 6, yl); ctx.stroke(); ctx.setLineDash([]);
  ctx.textAlign = 'center'; ctx.fillText("training steps", (W + p) / 2, H - 8);
  if (lossHist.length < 2) return;
  const xmax = Math.max(100, lossHist.length);
  ctx.strokeStyle = '#2b6cd4'; ctx.lineWidth = 2; ctx.beginPath();
  lossHist.forEach((l, k) => {
    const x = p + (W - 6 - p) * k / xmax, y = H - p - (H - p - 10) * Math.min(l, ymax) / ymax;
    if (k) ctx.lineTo(x, y); else ctx.moveTo(x, y);
  });
  ctx.stroke();
}

let selected = ALL.findIndex(p => p.unseen);
function buildPicker() {
  const box = document.getElementById('picker'); box.innerHTML = '';
  ALL.forEach((p, k) => {
    const b = document.createElement('button');
    b.type = 'button'; b.className = 'pick' + (k === selected ? ' sel' : '') + (p.unseen ? ' unseen' : '');
    b.title = p.unseen ? "Held-out image, never seen during training" : p.cap;
    b.setAttribute('aria-label', p.unseen ? "Held-out image, never seen during training" : p.cap);
    const cv = document.createElement('canvas'); cv.width = 36; cv.height = 36;
    drawShape(cv.getContext('2d'), p.s.key, p.c.css, 18, 18, 13);
    b.appendChild(cv);
    if (p.unseen) { const t = document.createElement('span'); t.className = 'badge'; t.textContent = "new"; b.appendChild(t); }
    b.addEventListener('click', () => { selected = k; buildPicker(); renderZeroShot(); });
    box.appendChild(b);
  });
}

function renderZeroShot() {
  const caps = document.getElementById('cands').value.split(/[,，、;\n]+/).map(t => t.trim()).filter(Boolean).slice(0, 12);
  const bars = document.getElementById('bars'); bars.innerHTML = '';
  if (!caps.length) return;
  const pr = zeroShot(ALL[selected].img, caps);
  const best = pr.indexOf(Math.max(...pr));
  caps.forEach((c, k) => {
    const row = document.createElement('div'); row.className = 'row' + (k === best ? ' best' : '');
    const t = document.createElement('span'); t.className = 't'; t.textContent = (k === best ? '▶ ' : '') + c; t.title = c;
    const tr = document.createElement('div'); tr.className = 'track';
    const f = document.createElement('div'); f.className = 'fill'; f.style.width = (100 * pr[k]).toFixed(1) + '%'; tr.appendChild(f);
    const v = document.createElement('span'); v.className = 'v'; v.textContent = (100 * pr[k]).toFixed(0) + '%';
    row.append(t, tr, v); bars.appendChild(row);
  });
  const seen = VOCAB.filter((w, k) => caps.some(c => norm(c).includes(VOCAB_N[k])));
  document.getElementById('tokens').textContent = i18nFmt("Words the text encoder recognised: %words%. Its whole vocabulary: %vocab%.", { words: seen.join(' · ') || '—', vocab: VOCAB.join(' · ') });
}

function refresh() {
  const F = forward(); lastF = F;
  let acc = 0;
  for (let i = 0; i < N; i++) if (F.rowP[i].indexOf(Math.max(...F.rowP[i])) === i) acc++;
  document.getElementById('stats').textContent = i18nFmt("step %step% · loss %loss% · τ = %tau% · correct matches %acc%/%n%", {
    step: stepCount, loss: F.loss.toFixed(3), tau: (1 / F.s).toFixed(3), acc: acc, n: N });
  drawHeat(F); drawLoss(); renderZeroShot();
}

function train(k) { for (let r = 0; r < k; r++) trainStep(); refresh(); }
function stopAuto() { if (autoTimer) { clearInterval(autoTimer); autoTimer = null; } document.getElementById('auto').classList.remove('on'); document.getElementById('auto').textContent = "Auto-train"; }
document.getElementById('step1').addEventListener('click', () => train(1));
document.getElementById('step25').addEventListener('click', () => train(25));
document.getElementById('auto').addEventListener('click', () => {
  if (autoTimer) { stopAuto(); return; }
  const b = document.getElementById('auto'); b.classList.add('on'); b.textContent = "Pause";
  autoTimer = setInterval(() => { train(3); if (stepCount >= 600) stopAuto(); }, 60);
});
document.getElementById('reset').addEventListener('click', () => { stopAuto(); reset(); });
document.getElementById('cands').addEventListener('input', renderZeroShot);

// Default candidates: every shape in green, plus two triangles of other colours.
const green = COLORS[1];
document.getElementById('cands').value = [
  ...SHAPES.map(s => caption(green, s)), caption(COLORS[0], SHAPES[2]), caption(COLORS[2], SHAPES[2])
].join('\n');
buildPicker();
reset();

Result

Console

Things to watch:

The matrix becomes diagonal. At first the similarities are random. After a few dozen steps each row (an image) lights up only in its own column (its caption), and the off-diagonal cells turn negative or near zero.
The loss falls from above $\ln 8 \approx 2.08$ to almost 0. $\ln N$ is the loss of a model that guesses uniformly; the initial value is often higher because the starting temperature of 0.07 amplifies random similarities.
The temperature drops. The model learns to sharpen its softmax as the matching becomes confident.
Zero-shot works on the held-out image. The green triangle was never in the batch, yet "green" was learned from the green circle and square, and "triangle" from the red and blue triangles. Because the encoders compose those pieces, the new caption usually wins. Press New random init a few times: occasionally it does not — a small reminder that generalisation is a tendency, not a guarantee.

The Real Complexity

The loss. For a batch of $N$ pairs let $u_i$ and $v_j$ be the unit-length image and text embeddings, and let the logits be

\ell_{ij} = e^{t}\, u_i^{\top} v_j ,

where $e^{t} = 1/\tau$ is a learned scale. CLIP minimises the average of two cross-entropies, one across each row and one down each column:

\mathcal{L} = \frac{1}{2N}\sum_{i=1}^{N}\left[-\log\frac{e^{\ell_{ii}}}{\sum_{j} e^{\ell_{ij}}} \;-\; \log\frac{e^{\ell_{ii}}}{\sum_{j} e^{\ell_{ji}}}\right].

Each term is an InfoNCE loss: the true caption must beat the $N-1$ wrong ones in the batch, and the true image must beat the $N-1$ wrong images. The gradient with respect to the logits is simply $\frac{1}{2N}\left(P^{\text{row}} + P^{\text{col}} - 2I\right)$ , where $P^{\text{row}}$ and $P^{\text{col}}$ are the row- and column-softmaxes — exactly what the demo back-propagates.

The temperature. $\tau$ is learned as a log-parameter, initialised to $0.07$ and clipped so that the logits are never multiplied by more than $100$ . Too high a temperature makes every pair look alike; too low makes training unstable.

The costs.

The similarity matrix of a batch costs $O(N^2 d)$ time and $O(N^2)$ memory. CLIP used $N = 32{,}768$ , so each step compares about $1.07 \times 10^{9}$ image–caption pairs; embeddings are gathered across all GPUs so every example sees every other example as a negative.
In practice the encoders dominate the compute. The paper reports 32 epochs over the dataset; the largest ResNet (RN50x64) took 18 days on 592 V100 GPUs and the largest ViT (ViT-L/14) 12 days on 256 V100 GPUs.
The objective was chosen for efficiency: in the paper's experiments, the contrastive objective learned zero-shot ImageNet accuracy roughly 4× faster than predicting the caption's bag of words, which in turn was about 3× faster than generating the caption word by word.

Zero-shot classification. To classify into $K$ classes, embed $K$ prompts such as "a photo of a {label}." and pick

\hat{y} = \arg\max_{k}\; u^{\top} v_k .

The text embeddings act as the weights of a linear classifier written in English. After the $K$ prompts are embedded once, each image costs one encoder pass plus $O(Kd)$ . The prompt matters: the paper found that "A photo of a {label}." alone added 1.3 points on ImageNet over the bare label, and averaging 80 prompt templates added another 3.5. The best model, ViT-L/14 at 336 px, reached 76.2 % zero-shot top-1 on ImageNet, matching the original supervised ResNet-50 without using any of ImageNet's 1.28 million labelled training images.

Robustness. On natural distribution shifts of ImageNet (sketches, renditions, ObjectNet, adversarially filtered images), zero-shot CLIP lost much less accuracy than supervised ImageNet models of equal ImageNet accuracy, closing up to 75 % of that "robustness gap". The authors are careful to note that adapting CLIP to ImageNet with supervised data erodes part of this advantage.

Known limits.

Counting and spatial relations: "three dogs" vs "two dogs", or "a red cube on a blue cube" vs the reverse, are weak spots, because a caption-matching objective rewards recognising what is present more than how it is arranged.
Fine-grained and abstract tasks: car models, aircraft variants and flower species are hard zero-shot; on MNIST digits zero-shot CLIP scores about 88 %, below a simple logistic regression on raw pixels — handwritten digits are rare in web captions.
Typographic attacks: in March 2021 OpenAI researchers showed that sticking a paper label reading "iPod" onto an apple makes CLIP call it an iPod — the model reads text in images very well, and trusts it too much.
Social bias: web captions carry stereotypes, and the paper's own audit found harmful misclassifications that depend heavily on which class names are offered.

The softmax is global; the sigmoid is not. In 2023 Zhai, Mustafa, Kolesnikov and Beyer (Google) proposed SigLIP (ICCV 2023), which replaces the softmax by an independent binary decision for every cell of the matrix:

\mathcal{L}_{\text{sig}} = -\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{N} \log \sigma\!\big(z_{ij}\,(e^{t}\,u_i^{\top}v_j + b)\big), \qquad z_{ij} = \begin{cases}+1 & i=j\\ -1 & i\neq j\end{cases}

with a learned bias $b$ . No row or column normalisation is needed, so the matrix can be processed in chunks without gathering all similarities in one place, and the loss performs better at small batch sizes. The authors found returns diminishing beyond a batch of about 32k.

Where It Matters

A shared space for pictures and words turned out to be a universal adapter:

Text-to-image generation. Stable Diffusion v1 conditions its diffusion model on the text encoder of a frozen CLIP ViT-L/14, and Stable Diffusion 2 switched to an OpenCLIP encoder. OpenAI's DALL·E 2 (2022, "unCLIP") goes further: it generates a CLIP image embedding from the caption and then decodes that embedding into pixels. Before diffusion took over, artists steered generators by following the gradient of CLIP similarity ("CLIP guidance"), a precursor of classifier-free guidance.
Search and retrieval. Embed a photo library once, then answer a text query by a nearest-neighbour search over the vectors — or search with an image instead. The same embeddings feed multimodal retrieval-augmented generation.
Building datasets. The open LAION datasets kept only web pairs whose CLIP image–text similarity passed a threshold, and those datasets in turn trained OpenCLIP and Stable Diffusion.
Eyes for language models. Many open multimodal LLMs bolt a CLIP-style vision encoder onto a language model — LLaVA (2023) used CLIP ViT-L/14, and Google's PaliGemma uses a SigLIP encoder.
Evaluation. CLIPScore (2021) rates how well a generated image or caption matches a text without a human reference.

The family tree.

ALIGN (Google, Jia et al., 2021) reached a similar result with about 1.8 billion noisy alt-text pairs and almost no filtering, showing that scale can compensate for messy data.
OpenCLIP (from 2021) is an open-source reproduction trained on the public LAION-400M and LAION-5B datasets, which made it possible to study CLIP's scaling laws openly. LAION-5B was withdrawn in December 2023 after the Stanford Internet Observatory found links to child sexual abuse material in it; a cleaned Re-LAION-5B was released in August 2024 — a sharp reminder that "scrape the web" is a data-governance decision, not just an engineering one.
SigLIP (2023) swapped the softmax for a sigmoid, and SigLIP 2 (Google DeepMind, February 2025) added captioning and self-supervised losses, multilingual data and better localisation.

The idea also echoes an older one: just as word2vec placed words with similar contexts close together, CLIP places an image close to the words people use to describe it. And CLIP's image towers are ordinary convolutional networks and Vision Transformers — what changed was the supervision, not the architecture.

Conclusion

CLIP's recipe fits on a napkin: two encoders, a normalised dot product, a learned temperature and a cross-entropy run in both directions over a batch. Scaled to hundreds of millions of captioned images, that napkin produced a visual model that could be pointed at new tasks simply by writing the classes, and an embedding space that later became the connective tissue of image generators, search engines and multimodal assistants.

It is not magic. CLIP inherits the blind spots of web captions — it counts badly, confuses fine-grained categories, believes the words written on objects and reflects the biases of its data. Its successors attack exactly these points: cleaner or larger data, the sigmoid loss, extra objectives for detail and localisation.

The lasting lesson is about supervision. A label says one thing about an image; a sentence can say anything. By learning to tell which sentence belongs to which picture, CLIP made language the interface to vision — and the toy above shows that the core of that idea is small enough to train in a browser tab.

Comments

Loading comments...

https://www.kipuhub.com/en/article/clip/Content licensed under CC BY-NC 4.0.

Cookies & privacy

Teaching images and words to share a space