Introduction

For thirty years, convolutional neural networks (CNNs) owned computer vision. They slide small filters across an image, progressively building up an understanding of edges, textures, and objects — the translation equivariance is baked right in.

In 2020 a team at Google Brain asked a radical question: what if you skipped the convolutions entirely? Their model, ViT (Vision Transformer), slices an image into a grid of square patches, flattens each patch into a vector, and feeds the whole sequence to a plain Transformer — the same architecture that already ruled natural language processing.

The result was striking: given enough training data, ViT matched and then beat the best CNNs on image benchmarks. The spatial bias that convolutions hard-coded turned out to be learnable from data, not a prerequisite. The border between vision and language modelling had quietly dissolved.

Try It: Image as Patches

A ViT's first step is turning a 2-D image into a 1-D sequence of patch tokens. The demo below lets you control the patch size and watch the image get sliced.

<!-- ViT patch-embedding demo — slices a synthetic scene into patch tokens -->
<div class="controls">
  <label for="patch-size">Patch size: <span id="ps-val">4</span>px</label>
  <input type="range" id="patch-size" min="2" max="8" step="2" value="4" title="Patch size:">
  <button id="btn-reset" type="button">Reset</button>
</div>
<div class="canvas-row">
  <canvas id="img-canvas" width="128" height="128" title="Original image"></canvas>
  <canvas id="patch-canvas" width="128" height="128" title="Patched image"></canvas>
</div>
<div class="labels-row">
  <span>Original image</span>
  <span>Patched image</span>
</div>
<div class="stat" id="stat"></div>
<div class="token-list" id="token-list"></div>

/* Vision Transformer patch demo styles */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
.controls { display: flex; align-items: center; gap: .6rem; flex-wrap: wrap; margin-bottom: .6rem; }
label { font-size: .85rem; color: #444; white-space: nowrap; }
input[type=range] { flex: 1; min-width: 80px; max-width: 160px; cursor: pointer; }
button { font: 600 13px system-ui; padding: .35rem .75rem; background: #1d3557; color: #fff;
         border: none; border-radius: 7px; cursor: pointer; }
button:hover { background: #2a4d7a; }
.canvas-row { display: flex; gap: 10px; align-items: flex-start; }
canvas { border: 1px solid #cdd9e3; border-radius: 6px; image-rendering: pixelated; }
.labels-row { display: flex; gap: 10px; font-size: .75rem; color: #666; margin-top: 2px; }
.labels-row span { width: 128px; text-align: center; }
.stat { font-size: .88rem; font-weight: 600; margin: .5rem 0 .3rem; color: #1d3557; }
.token-list { display: flex; flex-wrap: wrap; gap: 4px; max-height: 160px; overflow-y: auto; }
.token { width: 28px; height: 28px; border-radius: 4px; border: 1px solid #adb1b8;
         display: flex; align-items: center; justify-content: center;
         font: 600 9px ui-monospace, monospace; color: #fff; cursor: default;
         position: relative; }
.token:hover .tip { display: block; }
.tip { display: none; position: absolute; bottom: 110%; left: 50%; transform: translateX(-50%);
       background: #222; color: #fff; font-size: 10px; padding: 2px 5px; border-radius: 4px;
       white-space: nowrap; z-index: 10; pointer-events: none; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// ViT patch tokeniser: splits a 128x128 canvas image into NxN patches and shows each as a colour token
const SIZE = 128; // canvas is 128x128 pixels
const imgCanvas = document.getElementById('img-canvas');
const pCanvas   = document.getElementById('patch-canvas');
const slider    = document.getElementById('patch-size');
const psVal     = document.getElementById('ps-val');
const statEl    = document.getElementById('stat');
const tokenList = document.getElementById('token-list');
const ctxImg    = imgCanvas.getContext('2d');
const ctxP      = pCanvas.getContext('2d');

// Draw a simple synthetic scene (same function used for both canvases)
function drawScene(ctx) {
  ctx.fillStyle = '#e8f4fc';
  ctx.fillRect(0, 0, SIZE, SIZE);
  // sky gradient area
  ctx.fillStyle = '#74b9e8';
  ctx.fillRect(0, 0, SIZE, SIZE * 0.55);
  // sun circle
  ctx.fillStyle = '#f9c74f';
  ctx.beginPath(); ctx.arc(100, 25, 14, 0, Math.PI * 2); ctx.fill();
  // grass ground
  ctx.fillStyle = '#52b788';
  ctx.fillRect(0, SIZE * 0.55, SIZE, SIZE * 0.45);
  // house body
  ctx.fillStyle = '#e76f51';
  ctx.fillRect(28, 68, 52, 46);
  // roof triangle
  ctx.fillStyle = '#c1440e';
  ctx.beginPath(); ctx.moveTo(20, 68); ctx.lineTo(54, 42); ctx.lineTo(88, 68); ctx.closePath(); ctx.fill();
  // door
  ctx.fillStyle = '#6d4c41';
  ctx.fillRect(46, 88, 16, 26);
  // left and right windows
  ctx.fillStyle = '#aed6f1';
  ctx.fillRect(34, 76, 14, 12);
  ctx.fillRect(66, 76, 14, 12);
  // tree trunk
  ctx.fillStyle = '#795548';
  ctx.fillRect(100, 88, 8, 26);
  // tree crown
  ctx.fillStyle = '#2d6a4f';
  ctx.beginPath(); ctx.arc(104, 80, 18, 0, Math.PI * 2); ctx.fill();
}

// Compute the average RGB of a patch starting at (x,y) with side length ps
function avgColor(imageData, x, y, ps) {
  let r = 0, g = 0, b = 0, count = 0;
  for (let py = y; py < y + ps && py < SIZE; py++) {
    for (let px = x; px < x + ps && px < SIZE; px++) {
      const i = (py * SIZE + px) * 4;
      r += imageData.data[i]; g += imageData.data[i+1]; b += imageData.data[i+2];
      count++;
    }
  }
  return [Math.round(r/count), Math.round(g/count), Math.round(b/count)];
}

function toHex(r, g, b) {
  return '#' + [r,g,b].map(v => v.toString(16).padStart(2,'0')).join('');
}

// Redraw both canvases for the current patch size
function update() {
  const ps = parseInt(slider.value, 10);
  psVal.textContent = ps;

  // Draw original scene and capture pixel data
  drawScene(ctxImg);
  const imageData = ctxImg.getImageData(0, 0, SIZE, SIZE);

  // Draw scene with patch grid overlay
  ctxP.clearRect(0, 0, SIZE, SIZE);
  drawScene(ctxP);
  ctxP.strokeStyle = 'rgba(255,255,255,0.85)';
  ctxP.lineWidth = 1;

  const patches = [];
  const cols = Math.floor(SIZE / ps);
  const rows = Math.floor(SIZE / ps);

  for (let row = 0; row < rows; row++) {
    for (let col = 0; col < cols; col++) {
      const x = col * ps, y = row * ps;
      ctxP.strokeRect(x + 0.5, y + 0.5, ps, ps);
      const [r, g, b] = avgColor(imageData, x, y, ps);
      patches.push({ row, col, r, g, b, x, y, ps });
    }
  }

  const nTokens = patches.length;
  const dim = ps * ps * 3; // flatten patch: ps*ps pixels * 3 channels
  statEl.textContent = fmt('Tokens: %nTokens%  |  Patch vector dim: %dim%  |  Sequence length: %nTokens%', { nTokens: nTokens, dim: dim });

  // Render the token strip below the canvases
  tokenList.innerHTML = '';
  patches.forEach((p, idx) => {
    const hex = toHex(p.r, p.g, p.b);
    const div = document.createElement('div');
    div.className = 'token';
    div.style.background = hex;
    div.style.color = (p.r*0.299 + p.g*0.587 + p.b*0.114) > 140 ? '#333' : '#fff';
    div.textContent = idx;
    const tip = document.createElement('span');
    tip.className = 'tip';
    tip.textContent = fmt('Patch (%row%,%col%) rgb(%r%,%g%,%b%)', { row: p.row, col: p.col, r: p.r, g: p.g, b: p.b });
    div.appendChild(tip);
    tokenList.appendChild(div);
  });
}

slider.addEventListener('input', update);
document.getElementById('btn-reset').addEventListener('click', () => {
  slider.value = 4; // back to 4px default
  update();
});

update(); // initial render
Result
Console

Notice that a smaller patch size gives more tokens but preserves finer detail. A larger patch is coarser but cheaper — the sequence length (and therefore the quadratic attention cost) shrinks fast. The position embedding shown beside each patch is how the model remembers spatial order even though the Transformer itself is order-agnostic.

The Real Complexity

The trade-offs between ViT and CNNs run deep:

  • Inductive bias: a CNN is born knowing that nearby pixels relate and that the same filter works anywhere in the image. ViT has no such assumption — it must learn locality from data. That costs more training examples, but it also means ViT can learn long-range dependencies a CNN never would.
  • Attention is quadratic: the self-attention mechanism computes interactions between every pair of tokens, so cost scales as O(n2)O(n^2) in the number of patches. Halving the patch size quadruples the sequence length and multiplies compute by sixteen. This is why patch size is the central engineering knob in ViT.
  • Global from layer one: a CNN in layer one can only see a small receptive field; ViT's attention at layer one can already relate patch 1 to patch 196. That global view is why ViT excels at tasks requiring long-range reasoning.
  • Data hunger: CNNs generalise well from tens of thousands of examples. The original ViT paper required the 300-million-image JFT-300M dataset to beat ResNet; later work (DeiT, 2020) showed careful training could close the gap on ImageNet-scale data.

This is the same attention-cost tension explored in Transformers and attention: the price of sequence-level reasoning is quadratic time, and vision just made the sequences much longer.

Where It Matters

Once you treat images as token sequences, every advance in language modelling becomes available to vision:

  • Image classification: ViT and its descendants (DeiT, Swin Transformer, BEiT) set state-of-the-art records on ImageNet and medical imaging datasets.
  • Object detection and segmentation: models like ViTDet and SAM (Segment Anything Model) use ViT backbones to identify and outline objects with unprecedented generality.
  • Multimodal AI: CLIP, Flamingo, and GPT-4V pair a vision encoder built on ViT with a language model, enabling zero-shot image understanding — ask in words, see in images.
  • Video: video is just images in time; treating frames as patch sequences lets models attend across both space and time simultaneously.
  • Scientific imaging: pathology slides, satellite imagery, and astronomical photos all benefit from ViT's ability to reason globally — a tumour in one corner of a slide can inform the interpretation of a distant corner.

The patch-token idea has become so central that it connects directly back to Transformers and attention: the same self-attention engine that reads sentences now reads the visual world.

Conclusion

Vision Transformers made one bold bet: that the inductive biases of CNNs are not necessary, just convenient. By treating an image as a sequence of patch tokens and handing it to plain self-attention, ViT unlocked a unified architecture that now spans language, vision, audio, and beyond.

The cost is real — quadratic attention, data hunger, and engineering complexity — but the payoff is equally real: global context from the very first layer, seamless fusion with language models, and a research ecosystem that moves as one. When you use a multimodal AI today, there is a good chance a descendant of ViT is looking at the images for you.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/vision-transformers/Content licensed under CC BY-NC 4.0.