Introduction

Computers don't understand words — they understand numbers. For a long time the best we could do was give each word an arbitrary index: "cat" = 4271, "dog" = 4272. Those numbers carry no meaning; "cat" and "dog" are just as far apart as "cat" and "the".

In 2013 a team at Google led by Tomas Mikolov published a paper that changed this. Their idea: train a neural network to predict which words appear near each other in a large body of text. As a side effect of learning to predict context, the network discovers that words with similar contexts should point in similar directions. The resulting word vectors — also called word embeddings — make meaning geometric.

The payoff is startling. Once you have these vectors, you can do arithmetic on concepts. Subtract the vector for "man" from the vector for "king", add the vector for "woman", and you land near the vector for "queen". The famous equation king⃗−man⃗+woman⃗≈queen⃗\vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \approx \vec{\text{queen}} is not a trick — it falls out of the geometry automatically.

Try It: Analogy Explorer

The demo below uses a small set of pre-computed word vectors (dimension 10, trained on a toy corpus). Each word lives at a point in high-dimensional space; the demo projects the two most informative directions onto your screen.

<!-- Word analogy explorer — three selects for A, B, C; canvas shows the projected space -->
<p class="hint">Analogy arithmetic in action: pick three words A, B, C and compute A − B + C. The nearest word in the pre-trained vector space is the answer. Classic example: king − man + woman ≈ queen.</p>
<div class="controls">
  <div class="row">
    <label>Compute: A − B + C =</label>
    <div class="analogy-row">
      <select id="sel-a">options for word A</select>
      <span class="op">−</span>
      <select id="sel-b">options for word B</select>
      <span class="op">+</span>
      <select id="sel-c">options for word C</select>
      <span class="op">=</span>
      <span id="result-word" class="result-word">?</span>
    </div>
  </div>
  <div class="btns">
    <button id="btn-run" type="button">Compute analogy</button>
    <button id="btn-reset" type="button" class="ghost">Reset</button>
  </div>
</div>
<div id="status" class="status"></div>
<canvas id="canvas" width="420" height="240"></canvas>
<p class="caption">2D projection of the 10-dimensional word vectors (first two principal components).</p>

/* Styles for the word embedding analogy explorer */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; background: #fff; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .6rem; line-height: 1.5; }
.controls { margin-bottom: .5rem; }
.row { margin-bottom: .4rem; }
label { font-size: .82rem; font-weight: 600; color: #555; display: block; margin-bottom: .3rem; }
.analogy-row { display: flex; align-items: center; gap: .35rem; flex-wrap: wrap; }
select { font: 14px system-ui; padding: .3rem .5rem; border: 1px solid #ccc; border-radius: 6px; background: #f9f9f9; cursor: pointer; }
.op { font-weight: 700; font-size: 1.1rem; color: #1d3557; }
.result-word { font-weight: 700; font-size: 1.05rem; color: #0a7d33; min-width: 4rem; padding: .2rem .5rem; background: #e8f5ec; border-radius: 6px; text-align: center; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin-top: .5rem; }
button { font: 600 14px system-ui; padding: .4rem .85rem; border: 1px solid #1d3557; background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.status { font-size: .9rem; font-weight: 600; min-height: 1.3em; margin: .3rem 0 .4rem; color: #555; }
.status.ok { color: #0a7d33; }
.status.err { color: #c92f3c; }
canvas { border: 1px solid #dde3ea; border-radius: 8px; display: block; max-width: 100%; }
.caption { font-size: .78rem; color: #888; margin: .3rem 0 0; text-align: center; }

const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Pre-computed 10-dimensional word vectors (toy corpus, not real Word2Vec weights).
// Vectors are hand-tuned so that gender and royalty axes are clearly separated.
const VECS = {
  king:    [ 0.82, 0.61, 0.15,-0.40, 0.30, 0.55,-0.10, 0.20, 0.70,-0.25],
  queen:   [ 0.75, 0.70, 0.10,-0.35, 0.28,-0.48,-0.08, 0.18, 0.65,-0.20],
  man:     [ 0.60, 0.20,-0.15,-0.50, 0.40, 0.62,-0.05, 0.10, 0.25,-0.18],
  woman:   [ 0.55, 0.30,-0.12,-0.45, 0.38,-0.55,-0.03, 0.08, 0.22,-0.14],
  prince:  [ 0.80, 0.55, 0.18,-0.38, 0.32, 0.52,-0.12, 0.22, 0.68,-0.22],
  princess:[ 0.72, 0.65, 0.12,-0.33, 0.30,-0.46,-0.09, 0.20, 0.62,-0.18],
  paris:   [ 0.10, 0.15, 0.90, 0.75,-0.20, 0.05, 0.60,-0.30,-0.10, 0.40],
  france:  [ 0.05, 0.10, 0.85, 0.80,-0.18, 0.08, 0.55,-0.28,-0.08, 0.38],
  berlin:  [ 0.12, 0.18, 0.88, 0.70,-0.22, 0.06, 0.58,-0.32,-0.12, 0.42],
  germany: [ 0.08, 0.12, 0.83, 0.78,-0.20, 0.09, 0.53,-0.30,-0.09, 0.40],
  dog:     [-0.20, 0.55,-0.30, 0.30, 0.80,-0.10, 0.25, 0.60,-0.15, 0.20],
  cat:     [-0.18, 0.52,-0.28, 0.32, 0.78,-0.08, 0.23, 0.58,-0.13, 0.18],
  puppy:   [-0.22, 0.58,-0.32, 0.28, 0.82,-0.12, 0.27, 0.62,-0.17, 0.22],
  kitten:  [-0.19, 0.54,-0.29, 0.31, 0.79,-0.09, 0.24, 0.59,-0.14, 0.19],
};
const WORDS = Object.keys(VECS);
const DIM = VECS.king.length;

// Vector arithmetic and cosine similarity helpers.
function dot(a, b) { let s = 0; for (let i = 0; i < DIM; i++) s += a[i] * b[i]; return s; }
function norm(a) { return Math.sqrt(dot(a, a)); }
function cosine(a, b) { const d = norm(a) * norm(b); return d < 1e-9 ? 0 : dot(a, b) / d; }
function add(a, b) { return a.map((x, i) => x + b[i]); }
function sub(a, b) { return a.map((x, i) => x - b[i]); }

// Find the nearest word to a query vector, excluding the three input words.
function nearest(vec, exclude) {
  let best = null, bestSim = -2;
  for (const w of WORDS) {
    if (exclude.includes(w)) continue;
    const s = cosine(vec, VECS[w]);
    if (s > bestSim) { bestSim = s; best = w; }
  }
  return { word: best, sim: bestSim };
}

// Project 10-D vectors to 2D using the top two principal components (power iteration).
// Cached after the first call so re-draws are fast.
function project2D() {
  const pts = WORDS.map(w => VECS[w]);
  const mean = Array(DIM).fill(0);
  pts.forEach(p => p.forEach((v, i) => { mean[i] += v / pts.length; }));
  const centered = pts.map(p => p.map((v, i) => v - mean[i]));

  // Two rounds of power iteration to find the first two principal directions.
  let v1 = centered[0].slice();
  for (let iter = 0; iter < 60; iter++) {
    let proj = centered.map(p => dot(p, v1));
    let nv = Array(DIM).fill(0);
    proj.forEach((s, i) => centered[i].forEach((x, d) => { nv[d] += s * x; }));
    const n = norm(nv); v1 = nv.map(x => x / n);
  }
  const scores1 = centered.map(p => dot(p, v1));
  const residual = centered.map((p, i) => p.map((x, d) => x - scores1[i] * v1[d]));
  let v2 = residual[1].slice();
  for (let iter = 0; iter < 60; iter++) {
    let proj = residual.map(p => dot(p, v2));
    let nv = Array(DIM).fill(0);
    proj.forEach((s, i) => residual[i].forEach((x, d) => { nv[d] += s * x; }));
    const n = norm(nv); v2 = nv.map(x => x / n);
  }
  const scores2 = residual.map(p => dot(p, v2));
  return { scores1, scores2, v1, v2, mean };
}

let projection = null;
function getProj() { if (!projection) projection = project2D(); return projection; }

function projectWord(vec) {
  const { v1, v2, mean } = getProj();
  const c = vec.map((x, i) => x - mean[i]);
  return { x: dot(c, v1), y: dot(c, v2) };
}

// Draw all words on canvas; highlight analogy words and the result vector.
function draw(resultVec, resultWord, analogyWords) {
  const canvas = document.getElementById('canvas');
  const ctx = canvas.getContext('2d');
  const W = canvas.width, H = canvas.height;
  ctx.clearRect(0, 0, W, H);

  const { scores1, scores2 } = getProj();
  const allX = scores1, allY = scores2;
  const pad = 40;
  const minX = Math.min(...allX) - 0.1, maxX = Math.max(...allX) + 0.1;
  const minY = Math.min(...allY) - 0.1, maxY = Math.max(...allY) + 0.1;
  const toCanvasX = x => pad + (x - minX) / (maxX - minX) * (W - 2 * pad);
  const toCanvasY = y => H - pad - (y - minY) / (maxY - minY) * (H - 2 * pad);

  // Draw each word as a dot, bigger and darker for analogy words and the result.
  WORDS.forEach((w, i) => {
    const cx = toCanvasX(scores1[i]);
    const cy = toCanvasY(scores2[i]);
    const isAnalogy = analogyWords.includes(w);
    const isResult = w === resultWord;
    ctx.beginPath();
    ctx.arc(cx, cy, isAnalogy || isResult ? 6 : 3.5, 0, 2 * Math.PI);
    ctx.fillStyle = isResult ? '#0a7d33' : isAnalogy ? '#1d3557' : '#9fb2c8';
    ctx.fill();
    ctx.font = `${isAnalogy || isResult ? '700' : '400'} 11px system-ui`;
    ctx.fillStyle = isResult ? '#0a7d33' : isAnalogy ? '#1d3557' : '#888';
    ctx.fillText(w, cx + 8, cy + 4);
  });

  // Draw the raw result vector (before nearest-word lookup) as a red marker.
  if (resultVec) {
    const rp = projectWord(resultVec);
    const rx = toCanvasX(rp.x), ry = toCanvasY(rp.y);
    ctx.beginPath(); ctx.arc(rx, ry, 5, 0, 2 * Math.PI);
    ctx.fillStyle = '#e63946'; ctx.fill();
    ctx.font = '700 11px system-ui'; ctx.fillStyle = '#e63946';
    ctx.fillText('✦ ' + 'result vector', rx + 8, ry + 4);

    // Dashed line from word A to the result vector.
    if (analogyWords.length > 0) {
      const ap = projectWord(VECS[analogyWords[0]]);
      const ax = toCanvasX(ap.x), ay = toCanvasY(ap.y);
      ctx.beginPath(); ctx.moveTo(ax, ay); ctx.lineTo(rx, ry);
      ctx.strokeStyle = '#e6394680'; ctx.lineWidth = 1.5;
      ctx.setLineDash([4, 3]); ctx.stroke(); ctx.setLineDash([]);
    }
  }
}

// Populate each select element with all vocabulary words.
function populate(selId, defaultWord) {
  const sel = document.getElementById(selId);
  sel.innerHTML = '';
  WORDS.forEach(w => {
    const o = document.createElement('option');
    o.value = w; o.textContent = w;
    if (w === defaultWord) o.selected = true;
    sel.appendChild(o);
  });
}

populate('sel-a', 'king');
populate('sel-b', 'man');
populate('sel-c', 'woman');
draw(null, null, []);

document.getElementById('btn-run').addEventListener('click', () => {
  const a = document.getElementById('sel-a').value;
  const b = document.getElementById('sel-b').value;
  const c = document.getElementById('sel-c').value;
  // Analogy arithmetic: result = vec(A) - vec(B) + vec(C).
  const result = add(sub(VECS[a], VECS[b]), VECS[c]);
  const { word: ans, sim } = nearest(result, [a, b, c]);
  document.getElementById('result-word').textContent = ans;
  const pct = Math.round(sim * 100);
  document.getElementById('status').textContent = i18nFmt('%a% − %b% + %c% ≈ %ans% (cosine similarity %pct%%)', { a, b, c, ans, pct });
  document.getElementById('status').className = 'status ok';
  draw(result, ans, [a, b, c]);
});

document.getElementById('btn-reset').addEventListener('click', () => {
  document.getElementById('sel-a').value = 'king';
  document.getElementById('sel-b').value = 'man';
  document.getElementById('sel-c').value = 'woman';
  document.getElementById('result-word').textContent = '?';
  document.getElementById('status').textContent = '';
  document.getElementById('status').className = 'status';
  draw(null, null, []);
});
Result
Console

Select a starting word, an analogy pair, and watch the arithmetic play out in the projected plane. The answer word is whichever word in the vocabulary sits closest (by cosine similarity) to the result vector — usually the one you'd expect.

The Real Complexity

The Skip-gram model is deceptively simple. Given a corpus of text, slide a window of fixed size ww across every word. For each center word cc, predict every context word oo within distance ww:

max⁡∑t∑−w≤j≤w, j≠0log⁡P(wt+j∣wt)\max \sum_{t} \sum_{-w \le j \le w,\, j \ne 0} \log P(w_{t+j} \mid w_t)

P(o∣c)P(o \mid c) is computed with a softmax over all vocabulary words:

P(o∣c)=exp⁡(u⃗o⋅v⃗c)∑kexp⁡(u⃗k⋅v⃗c)P(o \mid c) = \frac{\exp(\vec{u}_o \cdot \vec{v}_c)}{\sum_{k} \exp(\vec{u}_k \cdot \vec{v}_c)}

Every word gets two vectors: v⃗\vec{v} (center role) and u⃗\vec{u} (context role). Gradient descent nudges vectors of co-occurring words closer together and pushes unrelated ones apart.

Why does analogy arithmetic work? Because the training signal is symmetric: if "king" and "queen" appear in similar contexts, their vectors point in similar directions. The difference king⃗−queen⃗\vec{\text{king}} - \vec{\text{queen}} captures the gender axis. Subtracting and adding along that axis is just vector arithmetic, but the geometry makes it feel like reasoning.

The practical problem is the softmax denominator — summing over all VV vocabulary words per gradient step. For V=106V = 10^6 that is too slow. The celebrated fix is negative sampling: instead of normalizing over all words, compare the target pair against kk randomly drawn "noise" words. This reduces the per-step cost from O(V)O(V) to O(k)O(k).

See also neural network training for the broader picture of how gradient descent shapes these representations, and PAC learning for the theoretical framing of what learning from examples can and cannot achieve.

Where It Matters

Dense word vectors are a foundational building block of modern language technology:

  • Search and information retrieval: semantic search ranks documents by meaning, not just keyword overlap. Two sentences can share no words and still match because their embeddings are nearby.
  • Machine translation: sequence-to-sequence models translate by mapping source embeddings through attention to target embeddings — the geometry of meaning transfers across languages.
  • Sentiment analysis and classification: a linear classifier on top of averaged word vectors captures surprisingly rich sentiment. The vectors do the heavy lifting.
  • Recommendation systems: "users who liked X also liked Y" can be recast as nearest-neighbor lookup in embedding space — the same geometry works for items, users, and queries.
  • Transformer language models: the input layer of every modern transformer (BERT, GPT, …) is still a learned embedding table — just trained jointly with the attention layers rather than separately.

The word-vector idea scaled beyond words too: sentence embeddings, graph embeddings, and code embeddings all follow the same recipe: define a "context" signal and let gradient descent build the geometry.

Conclusion

Word embeddings are one of the most elegant ideas in machine learning: train a network to predict neighbors, and as a free bonus you get a map of meaning. The map is imperfect — it encodes biases present in the training corpus, struggles with polysemy, and collapses when words are rare — but it is remarkably useful.

The analogy king⃗−man⃗+woman⃗≈queen⃗\vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \approx \vec{\text{queen}} is not just a party trick. It tells us that language has geometry — that the regularities humans perceive as analogy are encoded in the directions of co-occurrence statistics. A two-layer neural network, trained with nothing but next-word prediction, rediscovers that structure on its own.

That is the deeper lesson: you don't have to tell a model what "royalty" or "gender" mean. Given enough text and a simple objective, the geometry emerges by itself.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/word-embeddings-skipgram/Content licensed under CC BY-NC 4.0.