Introduction

A transformer reads a sentence as an unordered bag of tokens. Without extra help it cannot tell "dog bites man" from "man bites dog" — the self-attention mechanism treats both identically. So before computing anything, every model must answer one question: where does each token sit?

Early models simply added a fixed sinusoidal signal to each token's vector (Vaswani et al., 2017). Later work learned separate position embeddings from scratch. Both approaches paste position information into the vector before attention runs — they modify the content to carry the address.

Rotary Position Embeddings (RoPE), introduced by Su et al. in 2021, take a cleaner path: instead of adding a position signal, they rotate the query and key vectors by an angle proportional to their position in the sequence. The geometry of the rotation is chosen so that the dot product qm⋅knq_m \cdot k_n — the raw attention score between token mm and token nn — depends only on the difference m−nm - n, not on mm or nn individually. Relative distance falls out automatically, for free.

This elegance is not academic. RoPE powers LLaMA, Mistral, Gemma, and most of the open-weight language models built after 2023. Understanding it means understanding the positional heartbeat of today's most capable AI systems.

Try It

Place two tokens anywhere in the sequence and watch what happens to their attention score. Each token's query or key vector is rotated by an angle proportional to its position. The dot product of the rotated vectors — the raw attention score — depends only on how far apart the tokens are, not on their absolute positions.

<!-- Main app container with controls, canvas and stats -->
<div class="rope-app">
  <p class="intro-text">Each token's query/key vector is rotated by an angle proportional to its position. The dot product (attention score) depends only on the gap between positions — not on where they sit in the sequence.</p>
  <div class="controls-grid">
    <div class="control-group">
      <label for="posA">Token A position <span id="valA" class="val-badge">0</span></label>
      <input type="range" id="posA" min="0" max="15" value="2" step="1" aria-label="Token A position">
    </div>
    <div class="control-group">
      <label for="posB">Token B position <span id="valB" class="val-badge">5</span></label>
      <input type="range" id="posB" min="0" max="15" value="5" step="1" aria-label="Token B position">
    </div>
  </div>
  <!-- Canvas where rotated vectors are drawn -->
  <canvas id="ropeCanvas" width="480" height="220" aria-label="Vector diagram showing rotated query and key vectors"></canvas>
  <div class="stats-row">
    <div class="stat-box">
      <span class="stat-label">Gap |m−n|</span>
      <span class="stat-value" id="gapVal">3</span>
    </div>
    <div class="stat-box highlight">
      <span class="stat-label">Attention score</span>
      <span class="stat-value" id="dotVal">0.00</span>
    </div>
    <div class="stat-box">
      <span class="stat-label">Relative angle</span>
      <span class="stat-value" id="angleVal">0.00</span>
    </div>
  </div>
  <p class="insight" id="insightMsg"></p>
  <div class="btns">
    <button id="btnTogether" type="button">Move together</button>
    <button id="btnApart" type="button">Move apart</button>
    <button id="btnReset" type="button" class="ghost">Reset</button>
  </div>
</div>

/* Box-sizing reset for all elements */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; padding: .5rem; }
.rope-app { max-width: 500px; margin: 0 auto; }
.intro-text { font-size: .88rem; color: #444; margin: 0 0 .8rem; line-height: 1.5; }
/* Two-column grid for the position sliders */
.controls-grid { display: grid; grid-template-columns: 1fr 1fr; gap: .6rem; margin-bottom: .6rem; }
.control-group label { display: flex; align-items: center; gap: .4rem; font-size: .85rem; font-weight: 600; margin-bottom: .3rem; }
.val-badge { background: #1d3557; color: #fff; border-radius: 99px; padding: .05rem .45rem; font-size: .78rem; min-width: 1.6rem; text-align: center; }
input[type=range] { width: 100%; accent-color: #1d3557; }
/* Canvas styles: full width, light border, light background */
#ropeCanvas { display: block; width: 100%; border: 1px solid #cdd9e3; border-radius: 8px; background: #f8fafc; margin: .4rem 0; }
/* Stats row: three boxes showing gap, dot product and angle */
.stats-row { display: flex; gap: .5rem; flex-wrap: wrap; margin: .4rem 0; }
.stat-box { flex: 1; min-width: 90px; background: #e8eef3; border-radius: 8px; padding: .35rem .6rem; text-align: center; border: 1px solid #cdd9e3; }
.stat-box.highlight { background: #1d3557; color: #fff; border-color: #1d3557; }
.stat-box.highlight .stat-label { color: #aac0d8; }
.stat-label { display: block; font-size: .72rem; color: #6b8094; text-transform: uppercase; letter-spacing: .04em; }
.stat-value { display: block; font-size: 1.1rem; font-weight: 700; margin-top: .15rem; }
/* Insight message and button row styles */
.insight { font-size: .84rem; min-height: 1.3em; color: #1d3557; font-weight: 500; margin: .3rem 0 .6rem; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 13px system-ui, sans-serif; padding: .4rem .8rem; border: 1px solid #1d3557; background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// RoPE: rotate q by m*theta, k by n*theta. Then q_rot · k_rot = f(n-m only).
// This is the core property: dot product depends on relative position, not absolute.

const THETA = Math.PI / 10; // base rotation frequency per position step
const canvas = document.getElementById('ropeCanvas');
const ctx = canvas.getContext('2d');
const W = canvas.width, H = canvas.height;

// Fixed unit query and key vectors (same for every position — only rotation changes)
const Q = [0.85, 0.53];
const K = [0.60, 0.80];

// 2D rotation by angle: standard rotation matrix
function rotate(v, angle) {
  const c = Math.cos(angle), s = Math.sin(angle);
  return [c * v[0] - s * v[1], s * v[0] + c * v[1]];
}

// standard 2D dot product
function dot(a, b) { return a[0] * b[0] + a[1] * b[1]; }

function getPositions() {
  return {
    m: parseInt(document.getElementById('posA').value, 10),
    n: parseInt(document.getElementById('posB').value, 10)
  };
}

// Recompute and redraw everything when sliders change
function draw() {
  const { m, n } = getPositions();
  const qRot = rotate(Q, m * THETA);
  const kRot = rotate(K, n * THETA);
  const gap = Math.abs(n - m);
  const score = dot(qRot, kRot);
  const relAngle = gap * THETA;

  // Update all numeric readouts
  document.getElementById('valA').textContent = m;
  document.getElementById('valB').textContent = n;
  document.getElementById('gapVal').textContent = gap;
  document.getElementById('dotVal').textContent = score.toFixed(3);
  document.getElementById('angleVal').textContent = (relAngle * 180 / Math.PI).toFixed(1) + '°';

  // Insight message depends on relative gap
  const msg = gap === 0
    ? 'Same position — vectors align, score is maximum.'
    : (gap <= 3
      ? i18nFmt('Gap of %gap% — vectors are close, score is high.', { gap })
      : i18nFmt('Gap of %gap% — vectors have rotated apart, score drops.', { gap }));
  document.getElementById('insightMsg').textContent = msg;

  ctx.clearRect(0, 0, W, H);

  // Draw from origin at canvas center
  const cx = W / 2, cy = H / 2;
  const scale = 90;

  // Light axes for reference
  ctx.strokeStyle = '#dde4ea';
  ctx.lineWidth = 1;
  ctx.beginPath(); ctx.moveTo(cx - scale * 1.3, cy); ctx.lineTo(cx + scale * 1.3, cy); ctx.stroke();
  ctx.beginPath(); ctx.moveTo(cx, cy - scale * 1.3); ctx.lineTo(cx, cy + scale * 1.3); ctx.stroke();

  // Draw q and k rotated vectors with arrowheads and labels
  function drawVec(v, color, label) {
    const x = cx + v[0] * scale;
    const y = cy - v[1] * scale;
    ctx.strokeStyle = color;
    ctx.lineWidth = 2.4;
    ctx.beginPath(); ctx.moveTo(cx, cy); ctx.lineTo(x, y); ctx.stroke();
    // Draw arrowhead at tip of vector
    const angle = Math.atan2(-(y - cy), x - cx);
    const AL = 9, AW = 0.38;
    ctx.fillStyle = color;
    ctx.beginPath();
    ctx.moveTo(x, y);
    ctx.lineTo(x - AL * Math.cos(angle - AW), y + AL * Math.sin(angle - AW));
    ctx.lineTo(x - AL * Math.cos(angle + AW), y + AL * Math.sin(angle + AW));
    ctx.closePath(); ctx.fill();
    ctx.fillStyle = color;
    ctx.font = '700 13px system-ui, sans-serif';
    ctx.fillText(label, x + 8, y - 6);
  }

  drawVec(qRot, '#e63946', 'q (A)');
  drawVec(kRot, '#457b9d', 'k (B)');

  // Orange arc showing the angle between the two vectors
  const dotClamped = Math.max(-1, Math.min(1, score));
  const arcAngleQ = Math.atan2(qRot[1], qRot[0]);
  const arcAngleK = Math.atan2(kRot[1], kRot[0]);
  ctx.strokeStyle = '#f4a261';
  ctx.lineWidth = 2;
  ctx.setLineDash([4, 3]);
  ctx.beginPath();
  ctx.arc(cx, cy, scale * 0.38, -arcAngleQ, -arcAngleK, arcAngleQ > arcAngleK);
  ctx.stroke();
  ctx.setLineDash([]);

  // Draw a dot at the origin
  ctx.fillStyle = '#1d3557';
  ctx.beginPath(); ctx.arc(cx, cy, 4, 0, Math.PI * 2); ctx.fill();
}

document.getElementById('posA').addEventListener('input', draw);
document.getElementById('posB').addEventListener('input', draw);

// Preset: same position — score should be maximum
document.getElementById('btnTogether').addEventListener('click', () => {
  document.getElementById('posA').value = 4;
  document.getElementById('posB').value = 4;
  draw();
});

// Preset: wide gap — score drops
document.getElementById('btnApart').addEventListener('click', () => {
  document.getElementById('posA').value = 1;
  document.getElementById('posB').value = 10;
  draw();
});

// Reset to default positions
document.getElementById('btnReset').addEventListener('click', () => {
  document.getElementById('posA').value = 2;
  document.getElementById('posB').value = 5;
  draw();
});

draw();
Result
Console

Move both tokens together and the score stays constant. Move them apart and it drops. That is RoPE working: absolute position disappears from the score, relative distance is all that remains.

How RoPE Really Works

The key insight lives in two dimensions. Take a 2D vector v=(x,y)\mathbf{v} = (x, y) and rotate it by angle θ\theta:

R(θ) v=(cos⁡θ−sin⁡θsin⁡θcos⁡θ)(xy)R(\theta)\,\mathbf{v} = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix}

Now give token at position mm the rotation R(mθ)R(m\theta) and token at position nn the rotation R(nθ)R(n\theta). The dot product of the two rotated vectors satisfies:

R(mθ) q⋅R(nθ) k=q⋅R ⁣((n−m)θ) kR(m\theta)\,\mathbf{q} \cdot R(n\theta)\,\mathbf{k} = \mathbf{q} \cdot R\!\bigl((n - m)\theta\bigr)\,\mathbf{k}

because R(α)⊤R(β)=R(β−α)R(\alpha)^{\top} R(\beta) = R(\beta - \alpha). The absolute positions mm and nn vanish — only the gap n−mn - m survives. This is the entire RoPE trick.

In practice, query and key vectors have dd dimensions (typically 128 or more). RoPE pairs them up into d/2d/2 independent 2D planes and assigns each plane a different base frequency θi=10000−2i/d\theta_i = 10000^{-2i/d}. Low-frequency planes track long-range dependencies; high-frequency planes capture fine-grained local order — exactly the same multi-scale idea behind the original sinusoidal encoding, but now baked into the dot product instead of added to the input.

Why this beats additive encodings:

  • No extra parameters: the rotation is deterministic, not learned.
  • Relative position is free: models do not need to learn that "position 5 and position 8 are 3 apart"; the geometry guarantees it.
  • Generalizes beyond training length: the rotation can be extrapolated (with tricks like YaRN) to sequences much longer than those seen at training time.
  • Compatible with caching: rotated keys can be stored in the KV-cache and reused without recomputation.

The connection to the transformer attention mechanism is tight: RoPE modifies only the query and key projections, leaving values untouched. Everything else in the transformer stack — feed-forward layers, layer norms, the output projection — is oblivious to positions.

Where It Matters

The relative-position guarantee of RoPE has made it the dominant positional scheme in open-weight language models:

  • LLaMA and its descendants: Meta's LLaMA 2 and 3 series, Mistral, Mixtral, and Gemma all use RoPE. Any model fine-tuned on top of them inherits it.
  • Long-context extensions: techniques like YaRN (Yet Another RoPE extensioN) and LongRoPE rescale the rotation frequencies at inference time to extend context windows from 4k to 128k tokens or beyond, without full retraining.
  • Efficient KV-caching: because RoPE is applied to keys when they are written into the cache, rotated keys for past tokens need not be recomputed on each new token — the cache entry stays valid.
  • Multimodal models: vision-language models apply 2D variants of RoPE to image patch coordinates, letting the model reason about spatial relationships.

Compare this to absolute position embeddings: if a model trained with maximum length 2048 sees a 4096-token prompt, the embedding table has no entry — the model is lost. RoPE sidesteps the problem by design.

The deeper lesson is that neural network training does not always need learned tables for structure that mathematics can provide for free.

Conclusion

Rotary Position Embeddings distill a deep insight into a single rotation: encode position by how you rotate a vector, not by what you add to it. The dot product then automatically subtracts absolute positions and hands you only the relative gap — a mathematical gift that costs no parameters and generalizes beyond any fixed context length.

That gift is why RoPE now lives at the core of most open-weight language models. The next time you prompt a modern LLM and it tracks the thread across thousands of tokens, a tiny rotation matrix is quietly doing the bookkeeping — keeping every word exactly where it belongs in the sequence.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/rotary-position-embeddings/Content licensed under CC BY-NC 4.0.