Word Embeddings (Skip-gram)

Computers don't understand words — they understand numbers. For a long time the best we could do was give each word an arbitrary index: "cat" = 4271, "dog" = 4272. Those numbers carry no meaning; "cat" and "dog" are just as far apart as "cat" and "the".

In 2013 a team at Google led by Tomas Mikolov published a paper that changed this. Their idea: train a neural network to predict which words appear near each other in a large body of text. As a side effect of learning to predict context, the network discovers that words with similar contexts should point in similar directions. The resulting word vectors — also called word embeddings — make meaning geometric.

The payoff is startling. Once you have these vectors, you can do arithmetic on concepts. Subtract the vector for "man" from the vector for "king", add the vector for "woman", and you land near the vector for "queen". The famous equation $\vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \approx \vec{\text{queen}}$ is not a trick — it falls out of the geometry automatically.

The demo below uses a small set of pre-computed word vectors (dimension 10, trained on a toy corpus). Each word lives at a point in high-dimensional space; the demo projects the two most informative directions onto your screen.

<!-- {{c_html_intro}} -->
<p class="hint">{{hint_para}}</p>
<div class="controls">
  <div class="row">
    <label>{{lbl_analogy}}</label>
    <div class="analogy-row">
      <select id="sel-a">{{c_sel_a}}</select>
      <span class="op">−</span>
      <select id="sel-b">{{c_sel_b}}</select>
      <span class="op">+</span>
      <select id="sel-c">{{c_sel_c}}</select>
      <span class="op">=</span>
      <span id="result-word" class="result-word">?</span>
    </div>
  </div>
  <div class="btns">
    <button id="btn-run" type="button">{{btn_compute}}</button>
    <button id="btn-reset" type="button" class="ghost">{{btn_reset}}</button>
  </div>
</div>
<div id="status" class="status"></div>
<canvas id="canvas" width="420" height="240"></canvas>
<p class="caption">{{canvas_caption}}</p>

/* {{c_css_intro}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; background: #fff; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .6rem; line-height: 1.5; }
.controls { margin-bottom: .5rem; }
.row { margin-bottom: .4rem; }
label { font-size: .82rem; font-weight: 600; color: #555; display: block; margin-bottom: .3rem; }
.analogy-row { display: flex; align-items: center; gap: .35rem; flex-wrap: wrap; }
select { font: 14px system-ui; padding: .3rem .5rem; border: 1px solid #ccc; border-radius: 6px; background: #f9f9f9; cursor: pointer; }
.op { font-weight: 700; font-size: 1.1rem; color: #1d3557; }
.result-word { font-weight: 700; font-size: 1.05rem; color: #0a7d33; min-width: 4rem; padding: .2rem .5rem; background: #e8f5ec; border-radius: 6px; text-align: center; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin-top: .5rem; }
button { font: 600 14px system-ui; padding: .4rem .85rem; border: 1px solid #1d3557; background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.status { font-size: .9rem; font-weight: 600; min-height: 1.3em; margin: .3rem 0 .4rem; color: #555; }
.status.ok { color: #0a7d33; }
.status.err { color: #c92f3c; }
canvas { border: 1px solid #dde3ea; border-radius: 8px; display: block; max-width: 100%; }
.caption { font-size: .78rem; color: #888; margin: .3rem 0 0; text-align: center; }

// Code not found

Select a starting word, an analogy pair, and watch the arithmetic play out in the projected plane. The answer word is whichever word in the vocabulary sits closest (by cosine similarity) to the result vector — usually the one you'd expect.

The Skip-gram model is deceptively simple. Given a corpus of text, slide a window of fixed size $w$ across every word. For each center word $c$, predict every context word $o$ within distance $w$:

$$\max \sum_{t} \sum_{-w \le j \le w,\, j \ne 0} \log P(w_{t+j} \mid w_t)$$

$P(o \mid c)$ is computed with a softmax over all vocabulary words:

$$P(o \mid c) = \frac{\exp(\vec{u}_o \cdot \vec{v}_c)}{\sum_{k} \exp(\vec{u}_k \cdot \vec{v}_c)}$$

Every word gets two vectors: $\vec{v}$ (center role) and $\vec{u}$ (context role). Gradient descent nudges vectors of co-occurring words closer together and pushes unrelated ones apart.

Why does analogy arithmetic work? Because the training signal is symmetric: if "king" and "queen" appear in similar contexts, their vectors point in similar directions. The difference $\vec{\text{king}} - \vec{\text{queen}}$ captures the gender axis. Subtracting and adding along that axis is just vector arithmetic, but the geometry makes it feel like reasoning.

The practical problem is the softmax denominator — summing over all $V$ vocabulary words per gradient step. For $V = 10^6$ that is too slow. The celebrated fix is negative sampling: instead of normalizing over all words, compare the target pair against $k$ randomly drawn "noise" words. This reduces the per-step cost from $O(V)$ to $O(k)$.

See also neural network training for the broader picture of how gradient descent shapes these representations, and PAC learning for the theoretical framing of what learning from examples can and cannot achieve.

Dense word vectors are a foundational building block of modern language technology:

• Search and information retrieval: semantic search ranks documents by meaning, not just keyword overlap. Two sentences can share no words and still match because their embeddings are nearby.
• Machine translation: sequence-to-sequence models translate by mapping source embeddings through attention to target embeddings — the geometry of meaning transfers across languages.
• Sentiment analysis and classification: a linear classifier on top of averaged word vectors captures surprisingly rich sentiment. The vectors do the heavy lifting.
• Recommendation systems: "users who liked X also liked Y" can be recast as nearest-neighbor lookup in embedding space — the same geometry works for items, users, and queries.
• Transformer language models: the input layer of every modern transformer (BERT, GPT, …) is still a learned embedding table — just trained jointly with the attention layers rather than separately.

The word-vector idea scaled beyond words too: sentence embeddings, graph embeddings, and code embeddings all follow the same recipe: define a "context" signal and let gradient descent build the geometry.

Word embeddings are one of the most elegant ideas in machine learning: train a network to predict neighbors, and as a free bonus you get a map of meaning. The map is imperfect — it encodes biases present in the training corpus, struggles with polysemy, and collapses when words are rare — but it is remarkably useful.

The analogy $\vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \approx \vec{\text{queen}}$ is not just a party trick. It tells us that language has geometry — that the regularities humans perceive as analogy are encoded in the directions of co-occurrence statistics. A two-layer neural network, trained with nothing but next-word prediction, rediscovers that structure on its own.

That is the deeper lesson: you don't have to tell a model what "royalty" or "gender" mean. Given enough text and a simple objective, the geometry emerges by itself.