Transformers & Attention

Every chatbot you have ever used — ChatGPT, Claude, Gemini — runs on the same underlying architecture: the transformer, introduced in 2017 by Vaswani and colleagues at Google in a paper memorably titled "Attention Is All You Need."

Before it, language models read a sentence the way you might read with a tiny flashlight — one word at a time, trying to remember everything that came before. The transformer threw that away. Its core idea is attention: when the model processes a word, that word gets to look at every other word in the sentence at once and decide, individually, how much each one matters.

In "The animal didn't cross the street because it was tired," what does it refer to? A transformer answers by letting it attend strongly to animal and weakly to street. That single trick — words deciding what to focus on — is the engine under the entire modern AI boom.

Type a short sentence below, then click any word. The demo turns each word into a tiny vector and computes attention weights: how much your chosen word (the query) matches every other word (the keys). Brighter cells mean stronger focus.

<p class="hint">{{hint}}</p>
<input id="sentence" type="text" value="the animal did not cross the street because it was tired" />
<div class="btns">
  <button id="run" type="button">{{btn_run}}</button>
  <button id="reset" type="button" class="ghost">{{btn_reset}}</button>
</div>
<div id="words" class="words"></div>
<div class="status" id="status">{{status_pick}}</div>
<div id="bars" class="bars"></div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
input { width: 100%; font: 500 15px system-ui, sans-serif; padding: .5rem .6rem;
        border: 1px solid #adb1b8; border-radius: 8px; margin-bottom: .6rem; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin-bottom: .8rem; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.words { display: flex; flex-wrap: wrap; gap: .35rem; margin: .2rem 0 .8rem; }
.word { padding: .3rem .55rem; border-radius: 8px; background: #e8eef3; color: #1d3557;
        border: 1px solid #cdd9e3; cursor: pointer; font: 600 14px system-ui, sans-serif;
        user-select: none; transition: all .1s; }
.word:hover { background: #d6e1ea; }
.word.query { background: #1d3557; color: #fff; border-color: #14253c; }
.status { font-size: .95rem; font-weight: 600; margin: .4rem 0 .6rem; min-height: 1.3em; color: #1d3557; }
.bars { display: grid; gap: 4px; }
.row { display: grid; grid-template-columns: 110px 1fr 44px; align-items: center; gap: .5rem; }
.lbl { font: 600 13px system-ui, sans-serif; text-align: right; color: #333;
       overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.track { height: 18px; background: #eef1f4; border-radius: 5px; overflow: hidden; }
.fill { height: 100%; background: #e63946; border-radius: 5px; transition: width .25s; }
.pct { font: 600 12px ui-monospace, monospace; color: #555; }
.self .fill { background: #1d3557; }

// Code not found

This is a toy version, but the shape is exactly right. A real transformer does the same thing with learned vectors of hundreds of dimensions, stacked across dozens of layers and many parallel attention heads — each head free to focus on a different kind of relationship, like grammar, meaning, or position.

How does attention actually work, and what does it cost?

• Query, Key, Value. Each word is turned into three vectors. A word's query asks "what am I looking for?"; every word's key advertises "here is what I offer." The match between a query and a key (their dot product) becomes an attention score, squashed into weights by a softmax. Those weights then mix the value vectors into a new, context-aware representation of the word.
• It is exact, not undecidable or open. There is no hard mathematical mystery here — attention is a precise, fully understood formula. The transformer is an engineering breakthrough, not an unsolved problem.
• But it scales as $O(n^{2})$. To let every word attend to every other word, the model compares all pairs of words. For a sequence of n words that is roughly $n^{2}$ comparisons. Double the text and you quadruple the work — and the memory.
• That quadratic wall is the real frontier. It is why longer context windows are so costly, and why a whole research area (sparse attention, linear attention, FlashAttention) races to make the same idea cheaper.

So the difficulty in transformers is not "can we compute this?" — we can, exactly. It is the same flavor of question that runs through all of P vs NP: how does the cost grow as the input grows, and how clever can we be about it?

"Let each part of the input decide what else it should pay attention to" turned out to be a startlingly general idea:

• Language models: every GPT, Claude and Gemini is a stack of transformer layers; attention is what lets them track meaning across thousands of words.
• Translation and summarization: the original 2017 paper was about translation, where a word in one language must align with the right words in another.
• Beyond text: the same machinery folds proteins (AlphaFold), generates images, and recognizes speech — anywhere the relationships between pieces matter more than the pieces alone.
• Why scale helps: because attention is parallel, transformers train efficiently on enormous data, which is exactly what made today's giant models possible.

Underneath every one of these sits ordinary neural network training — gradient descent tuning billions of numbers — with attention as the architecture that decides what each of those numbers gets to look at.

Transformers hide a simple, beautiful idea behind an intimidating name: instead of reading a sentence word by word, let every word look at every other word and decide what to focus on. That single mechanism — query, key, value, softmax — is the architecture behind essentially every large language model in the world.

It is not an unsolved problem or a deep impossibility; it is a precise formula we understand completely. Its one stubborn limit is cost: comparing all pairs of words grows quadratically, which is why making attention cheaper is one of the liveliest races in computing. The next time a chatbot resolves what "it" refers to, picture the grid you just lit up — one word, quietly deciding where to look.