Vision Transformers (ViT)

For thirty years, convolutional neural networks (CNNs) owned computer vision. They slide small filters across an image, progressively building up an understanding of edges, textures, and objects — the translation equivariance is baked right in.

In 2020 a team at Google Brain asked a radical question: what if you skipped the convolutions entirely? Their model, ViT (Vision Transformer), slices an image into a grid of square patches, flattens each patch into a vector, and feeds the whole sequence to a plain Transformer — the same architecture that already ruled natural language processing.

The result was striking: given enough training data, ViT matched and then beat the best CNNs on image benchmarks. The spatial bias that convolutions hard-coded turned out to be learnable from data, not a prerequisite. The border between vision and language modelling had quietly dissolved.

A ViT's first step is turning a 2-D image into a 1-D sequence of patch tokens. The demo below lets you control the patch size and watch the image get sliced.

<!-- {{c_html_comment}} -->
<div class="controls">
  <label for="patch-size">{{label_patch_size}} <span id="ps-val">4</span>px</label>
  <input type="range" id="patch-size" min="2" max="8" step="2" value="4" title="{{label_patch_size}}">
  <button id="btn-reset" type="button">{{btn_reset}}</button>
</div>
<div class="canvas-row">
  <canvas id="img-canvas" width="128" height="128" title="{{title_original}}"></canvas>
  <canvas id="patch-canvas" width="128" height="128" title="{{title_patched}}"></canvas>
</div>
<div class="labels-row">
  <span>{{title_original}}</span>
  <span>{{title_patched}}</span>
</div>
<div class="stat" id="stat"></div>
<div class="token-list" id="token-list"></div>

/* {{c_css_comment}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
.controls { display: flex; align-items: center; gap: .6rem; flex-wrap: wrap; margin-bottom: .6rem; }
label { font-size: .85rem; color: #444; white-space: nowrap; }
input[type=range] { flex: 1; min-width: 80px; max-width: 160px; cursor: pointer; }
button { font: 600 13px system-ui; padding: .35rem .75rem; background: #1d3557; color: #fff;
         border: none; border-radius: 7px; cursor: pointer; }
button:hover { background: #2a4d7a; }
.canvas-row { display: flex; gap: 10px; align-items: flex-start; }
canvas { border: 1px solid #cdd9e3; border-radius: 6px; image-rendering: pixelated; }
.labels-row { display: flex; gap: 10px; font-size: .75rem; color: #666; margin-top: 2px; }
.labels-row span { width: 128px; text-align: center; }
.stat { font-size: .88rem; font-weight: 600; margin: .5rem 0 .3rem; color: #1d3557; }
.token-list { display: flex; flex-wrap: wrap; gap: 4px; max-height: 160px; overflow-y: auto; }
.token { width: 28px; height: 28px; border-radius: 4px; border: 1px solid #adb1b8;
         display: flex; align-items: center; justify-content: center;
         font: 600 9px ui-monospace, monospace; color: #fff; cursor: default;
         position: relative; }
.token:hover .tip { display: block; }
.tip { display: none; position: absolute; bottom: 110%; left: 50%; transform: translateX(-50%);
       background: #222; color: #fff; font-size: 10px; padding: 2px 5px; border-radius: 4px;
       white-space: nowrap; z-index: 10; pointer-events: none; }

// Code not found

Notice that a smaller patch size gives more tokens but preserves finer detail. A larger patch is coarser but cheaper — the sequence length (and therefore the quadratic attention cost) shrinks fast. The position embedding shown beside each patch is how the model remembers spatial order even though the Transformer itself is order-agnostic.

The trade-offs between ViT and CNNs run deep:

• Inductive bias: a CNN is born knowing that nearby pixels relate and that the same filter works anywhere in the image. ViT has no such assumption — it must learn locality from data. That costs more training examples, but it also means ViT can learn long-range dependencies a CNN never would.
• Attention is quadratic: the self-attention mechanism computes interactions between every pair of tokens, so cost scales as $O(n^2)$ in the number of patches. Halving the patch size quadruples the sequence length and multiplies compute by sixteen. This is why patch size is the central engineering knob in ViT.
• Global from layer one: a CNN in layer one can only see a small receptive field; ViT's attention at layer one can already relate patch 1 to patch 196. That global view is why ViT excels at tasks requiring long-range reasoning.
• Data hunger: CNNs generalise well from tens of thousands of examples. The original ViT paper required the 300-million-image JFT-300M dataset to beat ResNet; later work (DeiT, 2020) showed careful training could close the gap on ImageNet-scale data.

This is the same attention-cost tension explored in Transformers and attention: the price of sequence-level reasoning is quadratic time, and vision just made the sequences much longer.

Once you treat images as token sequences, every advance in language modelling becomes available to vision:

• Image classification: ViT and its descendants (DeiT, Swin Transformer, BEiT) set state-of-the-art records on ImageNet and medical imaging datasets.
• Object detection and segmentation: models like ViTDet and SAM (Segment Anything Model) use ViT backbones to identify and outline objects with unprecedented generality.
• Multimodal AI: CLIP, Flamingo, and GPT-4V pair a vision encoder built on ViT with a language model, enabling zero-shot image understanding — ask in words, see in images.
• Video: video is just images in time; treating frames as patch sequences lets models attend across both space and time simultaneously.
• Scientific imaging: pathology slides, satellite imagery, and astronomical photos all benefit from ViT's ability to reason globally — a tumour in one corner of a slide can inform the interpretation of a distant corner.

The patch-token idea has become so central that it connects directly back to Transformers and attention: the same self-attention engine that reads sentences now reads the visual world.

Vision Transformers made one bold bet: that the inductive biases of CNNs are not necessary, just convenient. By treating an image as a sequence of patch tokens and handing it to plain self-attention, ViT unlocked a unified architecture that now spans language, vision, audio, and beyond.

The cost is real — quadratic attention, data hunger, and engineering complexity — but the payoff is equally real: global context from the very first layer, seamless fusion with language models, and a research ecosystem that moves as one. When you use a multimodal AI today, there is a good chance a descendant of ViT is looking at the images for you.