Introduction

Every chatbot you have ever used — ChatGPT, Claude, Gemini — runs on the same underlying architecture: the transformer, introduced in 2017 by Vaswani and colleagues at Google in a paper memorably titled "Attention Is All You Need."

Before it, language models read a sentence the way you might read with a tiny flashlight — one word at a time, trying to remember everything that came before. The transformer threw that away. Its core idea is attention: when the model processes a word, that word gets to look at every other word in the sentence at once and decide, individually, how much each one matters.

In "The animal didn't cross the street because it was tired," what does it refer to? A transformer answers by letting it attend strongly to animal and weakly to street. That single trick — words deciding what to focus on — is the engine under the entire modern AI boom.

Try It: Watch Attention Light Up

Type a short sentence below, then click any word. The demo turns each word into a tiny vector and computes attention weights: how much your chosen word (the query) matches every other word (the keys). Brighter cells mean stronger focus.

<p class="hint">Type a short sentence and press <b>Run attention</b>. Then click any word: the bars show how strongly that word (the <b>query</b>) attends to every other word (the <b>keys</b>). This is a toy of the same query–key–value math inside every LLM.</p>
<input id="sentence" type="text" value="the animal did not cross the street because it was tired" />
<div class="btns">
  <button id="run" type="button">Run attention</button>
  <button id="reset" type="button" class="ghost">Reset sentence</button>
</div>
<div id="words" class="words"></div>
<div class="status" id="status">Pick a word above to see what it attends to.</div>
<div id="bars" class="bars"></div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
input { width: 100%; font: 500 15px system-ui, sans-serif; padding: .5rem .6rem;
        border: 1px solid #adb1b8; border-radius: 8px; margin-bottom: .6rem; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin-bottom: .8rem; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.words { display: flex; flex-wrap: wrap; gap: .35rem; margin: .2rem 0 .8rem; }
.word { padding: .3rem .55rem; border-radius: 8px; background: #e8eef3; color: #1d3557;
        border: 1px solid #cdd9e3; cursor: pointer; font: 600 14px system-ui, sans-serif;
        user-select: none; transition: all .1s; }
.word:hover { background: #d6e1ea; }
.word.query { background: #1d3557; color: #fff; border-color: #14253c; }
.status { font-size: .95rem; font-weight: 600; margin: .4rem 0 .6rem; min-height: 1.3em; color: #1d3557; }
.bars { display: grid; gap: 4px; }
.row { display: grid; grid-template-columns: 110px 1fr 44px; align-items: center; gap: .5rem; }
.lbl { font: 600 13px system-ui, sans-serif; text-align: right; color: #333;
       overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.track { height: 18px; background: #eef1f4; border-radius: 5px; overflow: hidden; }
.fill { height: 100%; background: #e63946; border-radius: 5px; transition: width .25s; }
.pct { font: 600 12px ui-monospace, monospace; color: #555; }
.self .fill { background: #1d3557; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Toy self-attention. Each word -> a small fixed "embedding" vector (hash-based, so it is deterministic). For a chosen query word q we score every word k by the dot product q·k (scaled), then softmax the scores into attention weights that sum to 1 — exactly the shape of real scaled dot-product attention, just tiny.
const D = 8; // embedding dimension
let words = [], vecs = [], queryIdx = null;

// Deterministic pseudo-embedding for a token: stable across runs, no imports.
function embed(tok){
  const v = new Array(D).fill(0);
  for(let i=0;i<tok.length;i++){
    const c = tok.charCodeAt(i);
    for(let d=0; d<D; d++){
      // mix char code, position and dimension into a bounded value
      v[d] += Math.sin((c * (d+1) * 0.13) + (i+1) * 0.7 + d);
    }
  }
  // L2-normalize so dot products behave like cosine similarities in [-1,1]
  const norm = Math.hypot(...v) || 1;
  return v.map(x => x / norm);
}
function dot(a,b){ let s=0; for(let i=0;i<a.length;i++) s += a[i]*b[i]; return s; }
function softmax(xs){
  const m = Math.max(...xs);
  const ex = xs.map(x => Math.exp(x - m));
  const sum = ex.reduce((a,b)=>a+b, 0);
  return ex.map(e => e / sum);
}
// Attention weights of query word i over all words. Self-match is boosted a bit (real Q/K projections differ, but a word's strong self-affinity is typical).
function attention(i){
  const scale = Math.sqrt(D);
  const scores = vecs.map((k, j) => (dot(vecs[i], k) + (i===j ? 0.6 : 0)) * 4 / scale);
  return softmax(scores);
}

function tokenize(s){
  return s.toLowerCase().replace(/[^a-záéíóúñ0-9 ]/gi,' ').split(/\s+/).filter(Boolean).slice(0, 14);
}
function setStatus(t){ document.getElementById('status').textContent = t; }

function run(){
  words = tokenize(document.getElementById('sentence').value);
  vecs = words.map(embed);
  queryIdx = null;
  renderWords();
  document.getElementById('bars').innerHTML = '';
  setStatus(words.length ? 'Pick a word above to see what it attends to.' : 'Type a sentence first.');
}
function renderWords(){
  const w = document.getElementById('words'); w.innerHTML = '';
  words.forEach((tok, i) => {
    const el = document.createElement('span');
    el.className = 'word' + (i===queryIdx ? ' query' : '');
    el.textContent = tok;
    el.addEventListener('click', () => select(i));
    w.appendChild(el);
  });
}
function select(i){
  queryIdx = i; renderWords();
  const a = attention(i);
  const order = a.map((w,j)=>[j,w]).sort((p,q)=>q[1]-p[1]);
  const top = order.find(([j])=>j!==i);
  setStatus(fmt('"%v%" attends most to "%v2%" (%round%%).', { v: words[i], v2: words[top[0]], round: Math.round(top[1]*100) }));
  const bars = document.getElementById('bars'); bars.innerHTML = '';
  order.forEach(([j, weight]) => {
    const row = document.createElement('div');
    row.className = 'row' + (j===i ? ' self' : '');
    const pct = Math.round(weight*100);
    row.innerHTML =
      '<div class="lbl">' + words[j] + '</div>' +
      '<div class="track"><div class="fill" style="width:' + (weight*100).toFixed(1) + '%"></div></div>' +
      '<div class="pct">' + pct + '%</div>';
    bars.appendChild(row);
  });
  console.log('query:', words[i], '| weights:', a.map(x=>x.toFixed(3)).join(' '),
              '| sum:', a.reduce((s,x)=>s+x,0).toFixed(4));
}
document.getElementById('run').addEventListener('click', run);
document.getElementById('reset').addEventListener('click', () => {
  document.getElementById('sentence').value = 'the animal did not cross the street because it was tired';
  run();
});
run();
Result
Console

This is a toy version, but the shape is exactly right. A real transformer does the same thing with learned vectors of hundreds of dimensions, stacked across dozens of layers and many parallel attention heads — each head free to focus on a different kind of relationship, like grammar, meaning, or position.

The Real Complexity

How does attention actually work, and what does it cost?

  • Query, Key, Value. Each word is turned into three vectors. A word's query asks "what am I looking for?"; every word's key advertises "here is what I offer." The match between a query and a key (their dot product) becomes an attention score, squashed into weights by a softmax. Those weights then mix the value vectors into a new, context-aware representation of the word.
  • It is exact, not undecidable or open. There is no hard mathematical mystery here — attention is a precise, fully understood formula. The transformer is an engineering breakthrough, not an unsolved problem.
  • But it scales as O(n2)O(n^{2}). To let every word attend to every other word, the model compares all pairs of words. For a sequence of n words that is roughly n2n^{2} comparisons. Double the text and you quadruple the work — and the memory.
  • That quadratic wall is the real frontier. It is why longer context windows are so costly, and why a whole research area (sparse attention, linear attention, FlashAttention) races to make the same idea cheaper.

So the difficulty in transformers is not "can we compute this?" — we can, exactly. It is the same flavor of question that runs through all of P vs NP: how does the cost grow as the input grows, and how clever can we be about it?

Where It Matters

"Let each part of the input decide what else it should pay attention to" turned out to be a startlingly general idea:

  • Language models: every GPT, Claude and Gemini is a stack of transformer layers; attention is what lets them track meaning across thousands of words.
  • Translation and summarization: the original 2017 paper was about translation, where a word in one language must align with the right words in another.
  • Beyond text: the same machinery folds proteins (AlphaFold), generates images, and recognizes speech — anywhere the relationships between pieces matter more than the pieces alone.
  • Why scale helps: because attention is parallel, transformers train efficiently on enormous data, which is exactly what made today's giant models possible.

Underneath every one of these sits ordinary neural network training — gradient descent tuning billions of numbers — with attention as the architecture that decides what each of those numbers gets to look at.

Conclusion

Transformers hide a simple, beautiful idea behind an intimidating name: instead of reading a sentence word by word, let every word look at every other word and decide what to focus on. That single mechanism — query, key, value, softmax — is the architecture behind essentially every large language model in the world.

It is not an unsolved problem or a deep impossibility; it is a precise formula we understand completely. Its one stubborn limit is cost: comparing all pairs of words grows quadratically, which is why making attention cheaper is one of the liveliest races in computing. The next time a chatbot resolves what "it" refers to, picture the grid you just lit up — one word, quietly deciding where to look.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/transformers-attention/Content licensed under CC BY-NC 4.0.