Introduction

Training a model like GPT-3 from scratch costs tens of millions of dollars. But what if you just want to teach the same model to write legal summaries, answer medical questions, or speak a new language? Full fine-tuning reruns the whole optimization over billions of weights — most of which barely need to move at all.

LoRA (Low-Rank Adaptation, Hu et al., 2021) makes a simple but powerful observation: the change in a weight matrix during fine-tuning lies on a very low-dimensional surface. Instead of updating the full matrix W∈Rd×kW \in \mathbb{R}^{d \times k}, LoRA freezes WW and injects two small matrices B∈Rd×rB \in \mathbb{R}^{d \times r} and A∈Rr×kA \in \mathbb{R}^{r \times k} (where r≪d,kr \ll d, k). Only BB and AA are trained; the effective update is ΔW=BA\Delta W = BA.

With a rank as low as r=4r = 4, LoRA reduces the trainable parameters for a large transformer layer from tens of millions to a few thousand — yet matches full fine-tuning on most benchmarks. The frozen weights cost no gradient memory, and the adapters can be swapped instantly between tasks.

LoRA belongs to a family called PEFT (Parameter-Efficient Fine-Tuning), which also includes prompt tuning, prefix tuning, and adapter layers. All share the same goal: adapt a frozen giant with a tiny trainable patch.

Try It: Low-Rank Update

A weight matrix WW is fixed (frozen). LoRA trains only two small matrices BB and AA whose product ΔW=BA\Delta W = BA approximates the desired change. Drag the rank slider to see how many parameters you actually train versus the full matrix, and watch the low-rank approximation fill in.

<!-- LoRA demo: visualize frozen W, low-rank delta W = B*A, and adapted W + delta W -->
<div class="demo-wrap">
  <div class="controls">
    <label for="rank-slider">Rank r = <strong id="rank-val">4</strong>
      &nbsp;<span class="dim-label">(d=<span id="lbl-d">8</span>, k=<span id="lbl-k">8</span>)</span>
    </label>
    <input id="rank-slider" type="range" min="1" max="8" value="4" step="1">
  </div>
  <div class="param-row">
    <div class="param-box full-box">
      <div class="param-title">Full params</div>
      <div class="param-num" id="full-params">64</div>
      <div class="param-sub">d × k</div>
    </div>
    <div class="param-box lora-box">
      <div class="param-title">LoRA params</div>
      <div class="param-num" id="lora-params">64</div>
      <div class="param-sub">r(d + k)</div>
    </div>
    <div class="param-box save-box">
      <div class="param-title">Reduction</div>
      <div class="param-num" id="reduction">1×</div>
      <div class="param-sub">fewer params</div>
    </div>
  </div>
  <div class="matrix-section">
    <div class="mat-wrap">
      <div class="mat-title">W (frozen)</div>
      <canvas id="canvas-W" width="128" height="128"></canvas>
    </div>
    <div class="mat-wrap">
      <div class="mat-title">ΔW = B·A (trained)</div>
      <canvas id="canvas-DW" width="128" height="128"></canvas>
    </div>
    <div class="mat-wrap">
      <div class="mat-title">W + ΔW (adapted)</div>
      <canvas id="canvas-WP" width="128" height="128"></canvas>
    </div>
  </div>
  <div class="error-row">
    <span>Relative approx. error: </span><strong id="approx-err">—</strong>
    &nbsp;|&nbsp;
    <span>Rank: </span><strong id="rank-label">—</strong>
  </div>
  <div class="legend">
    <span class="leg-neg">negative</span>
    <span class="leg-zero">0</span>
    <span class="leg-pos">positive</span>
  </div>
</div>

/* Styles for LoRA low-rank update visualizer */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; background: transparent; }
.demo-wrap { padding: .6rem .4rem; max-width: 520px; margin: 0 auto; }
.controls { margin-bottom: .6rem; }
label { font-size: .9rem; color: #444; display: block; margin-bottom: .3rem; }
.dim-label { font-size: .8rem; color: #777; }
input[type=range] { width: 100%; accent-color: #1d3557; }
.param-row { display: flex; gap: .5rem; margin-bottom: .8rem; }
.param-box { flex: 1; border-radius: 8px; padding: .5rem .4rem; text-align: center; border: 1px solid #cdd9e3; }
.full-box { background: #e8eef3; }
.lora-box { background: #d4edda; }
.save-box { background: #fff3cd; }
.param-title { font-size: .72rem; color: #555; margin-bottom: .15rem; }
.param-num { font-size: 1.3rem; font-weight: 700; color: #1d3557; }
.param-sub { font-size: .68rem; color: #888; }
.matrix-section { display: flex; gap: .6rem; justify-content: center; margin-bottom: .5rem; flex-wrap: wrap; }
.mat-wrap { text-align: center; }
.mat-title { font-size: .75rem; color: #444; margin-bottom: .2rem; }
canvas { border: 1px solid #cdd9e3; border-radius: 4px; image-rendering: pixelated; display: block; }
.error-row { font-size: .82rem; color: #444; margin-bottom: .4rem; }
.legend { display: flex; gap: .5rem; font-size: .75rem; align-items: center; }
.leg-neg { background: #c92f3c; color: #fff; padding: 1px 6px; border-radius: 3px; }
.leg-zero { background: #e8eef3; padding: 1px 6px; border-radius: 3px; }
.leg-pos { background: #0a7d33; color: #fff; padding: 1px 6px; border-radius: 3px; }

// LoRA low-rank update demo: approximate delta_W = B*A using truncated SVD of the true delta
const D = 8, K = 8; // weight matrix dimensions D x K

// Target weight matrix W* (what full fine-tuning would converge to)
const TARGET = [
  [ 0.8, -0.3,  0.5,  0.1, -0.6,  0.2,  0.4, -0.1],
  [-0.2,  0.7, -0.4,  0.6,  0.3, -0.5,  0.1,  0.8],
  [ 0.5,  0.1,  0.9, -0.7,  0.2,  0.4, -0.3,  0.6],
  [-0.3,  0.6, -0.2,  0.8, -0.5,  0.1,  0.7, -0.4],
  [ 0.4, -0.8,  0.3, -0.1,  0.9, -0.6,  0.2,  0.5],
  [-0.6,  0.2,  0.7, -0.4,  0.1,  0.8, -0.3,  0.3],
  [ 0.1,  0.5, -0.6,  0.3, -0.4,  0.7,  0.8, -0.2],
  [ 0.7, -0.1,  0.2,  0.5, -0.7,  0.3, -0.5,  0.9]
];

// Frozen base weight W (pre-trained, slightly perturbed from target)
const W = TARGET.map(row => row.map(v => v * 0.6 + (Math.random() - 0.5) * 0.1));

// Truncated SVD helpers to compute the rank-r approximation of delta W
function matMul(A, B) {
  const rows = A.length, inner = B.length, cols = B[0].length;
  const C = Array.from({length: rows}, () => new Array(cols).fill(0));
  for (let i = 0; i < rows; i++)
    for (let k = 0; k < inner; k++)
      for (let j = 0; j < cols; j++)
        C[i][j] += A[i][k] * B[k][j];
  return C;
}

function powerIter(M, iters) {
  // power iteration to find the top singular vector of M
  let v = Array.from({length: M[0].length}, () => Math.random() - 0.5);
  for (let t = 0; t < iters; t++) {
    const Mv = M.map(row => row.reduce((s, x, j) => s + x * v[j], 0));
    const MTMv = M[0].map((_, j) => M.reduce((s, row) => s + row[j] * Mv[M.indexOf(row)], 0));
    const norm = Math.sqrt(MTMv.reduce((s, x) => s + x * x, 0));
    v = norm > 1e-10 ? MTMv.map(x => x / norm) : v;
  }
  return v;
}

function deflate(M, u, s, v) {
  // remove the contribution of the found component
  return M.map((row, i) => row.map((x, j) => x - s * u[i] * v[j]));
}

function approxSVD(M, rank) {
  // compute rank-r SVD approximation of M
  let R = M.map(r => [...r]);
  const Us = [], Ss = [], Vs = [];
  for (let r = 0; r < rank; r++) {
    const vv = powerIter(R, 60);
    const Mv = R.map(row => row.reduce((s, x, j) => s + x * vv[j], 0));
    const sigma = Math.sqrt(Mv.reduce((s, x) => s + x * x, 0));
    if (sigma < 1e-10) break;
    const uu = Mv.map(x => x / sigma);
    Us.push(uu); Ss.push(sigma); Vs.push(vv);
    R = deflate(R, uu, sigma, vv);
  }
  return { Us, Ss, Vs };
}

function buildDeltaW(M, rank) {
  // build the rank-r approximation of the true delta W = W* - W
  const DELTA = TARGET.map((row, i) => row.map((v, j) => v - M[i][j]));
  const { Us, Ss, Vs } = approxSVD(DELTA, rank);
  const approx = Array.from({length: D}, () => new Array(K).fill(0));
  for (let r = 0; r < Us.length; r++)
    for (let i = 0; i < D; i++)
      for (let j = 0; j < K; j++)
        approx[i][j] += Ss[r] * Us[r][i] * Vs[r][j];
  return approx;
}

function frobErr(A, B) {
  // Frobenius norm of the residual
  let s = 0;
  for (let i = 0; i < A.length; i++)
    for (let j = 0; j < A[0].length; j++) {
      const d = A[i][j] - B[i][j]; s += d * d;
    }
  return Math.sqrt(s);
}

// draw a matrix as a heatmap on the given canvas, range [-vmax, vmax]
function drawMatrix(canvasId, M, vmax) {
  const canvas = document.getElementById(canvasId);
  const ctx = canvas.getContext('2d');
  const cw = canvas.width, ch = canvas.height;
  const cellW = cw / K, cellH = ch / D;
  for (let i = 0; i < D; i++) {
    for (let j = 0; j < K; j++) {
      const v = Math.max(-vmax, Math.min(vmax, M[i][j]));
      const t = v / vmax;
      let r, g, b;
      if (t < 0) { r = Math.round(201 * (-t)); g = Math.round(47 * (-t)); b = Math.round(60 * (-t)); r = 232 + Math.round((r - 232) * (-t)); g = 238 + Math.round((g - 238) * (-t)); b = 243 + Math.round((b - 243) * (-t)); }
      else { r = Math.round(232 + (10 - 232) * t); g = Math.round(238 + (125 - 238) * t); b = Math.round(243 + (51 - 243) * t); }
      ctx.fillStyle = `rgb(${r},${g},${b})`;
      ctx.fillRect(j * cellW, i * cellH, cellW, cellH);
    }
  }
}

function update() {
  const rank = parseInt(document.getElementById('rank-slider').value, 10);
  document.getElementById('rank-val').textContent = rank;
  document.getElementById('lbl-d').textContent = D;
  document.getElementById('lbl-k').textContent = K;

  const loraParams = rank * (D + K);
  const fullParams = D * K;
  document.getElementById('full-params').textContent = fullParams;
  document.getElementById('lora-params').textContent = loraParams;
  const ratio = fullParams / loraParams;
  document.getElementById('reduction').textContent = ratio.toFixed(1) + '×';
  document.getElementById('rank-label').textContent = 'r = ' + rank;

  const DW = buildDeltaW(W, rank);
  const WP = W.map((row, i) => row.map((v, j) => v + DW[i][j]));

  // relative error: how much of the delta is not captured by the rank-r approximation
  const fullDW = TARGET.map((row, i) => row.map((v, j) => v - W[i][j]));
  const err = frobErr(WP, TARGET) / frobErr(fullDW, Array.from({length: D}, () => new Array(K).fill(0)));
  document.getElementById('approx-err').textContent = (err * 100).toFixed(1) + '%';

  drawMatrix('canvas-W', W, 1.0);
  drawMatrix('canvas-DW', DW, 0.5);
  drawMatrix('canvas-WP', WP, 1.0);
}

document.getElementById('rank-slider').addEventListener('input', update);
update();
Result
Console

Notice the trade-off: at rank 1 you train almost nothing but the approximation is coarse; as rank approaches the full dimension dd you recover full fine-tuning at full cost. In practice, r∈{4,8,16}r \in \{4, 8, 16\} captures most of what a task needs while keeping the parameter count tiny.

The Math Behind LoRA

Why does a rank-4 update capture most of what fine-tuning needs? The answer is the intrinsic dimensionality hypothesis.

Aghajanyan et al. (2021) showed empirically that pre-trained language models can be fine-tuned by optimizing in a random low-dimensional subspace of the full parameter space. The model has learned such a rich general representation that adapting it to a specific task only requires movement along a handful of directions — not a full re-sculpting of all billions of weights.

LoRA makes this precise for a single weight matrix:

  • Full update: W′=W+ΔWW' = W + \Delta W, train d×kd \times k parameters.
  • LoRA update: ΔW=BA\Delta W = BA, train only r(d+k)r(d + k) parameters, where r≪min⁡(d,k)r \ll \min(d, k).
  • At inference time: merge W′=W+BAW' = W + BA (no runtime overhead) or keep B,AB, A separate for instant task-switching.

The rank rr is a hyperparameter that controls the expressivity-efficiency trade-off. For very large models (d=4096d = 4096, k=4096k = 4096), even r=8r = 8 trains 8×8192=65 5368 \times 8192 = 65\,536 parameters versus 16 777 21616\,777\,216 for the full matrix — a 256× reduction.

The initialization matters too: BB starts at zero (so ΔW=BA=0\Delta W = BA = 0 at step 0, identical to the pre-trained model), and AA is drawn from a Gaussian. This guarantees the adapted model starts exactly where the pre-trained model left off.

LoRA is now the backbone of most open-source fine-tuning pipelines for models like LLaMA, Mistral, and GPT-style architectures. See also how Transformers & Attention create the weight matrices that LoRA adapts.

Where It Matters

LoRA and PEFT techniques power nearly every practical large-model deployment today:

  • Instruction tuning: teaching a base model to follow chat instructions (ChatGPT-style RLHF pipelines use LoRA for the SFT step) without touching the billions of base weights.
  • Domain adaptation: a single base model plus a 10 MB LoRA adapter becomes a legal-document analyst, a medical-record coder, or a customer-service bot — all without separate full-size copies.
  • Multi-task serving: a single GPU can serve dozens of tasks by hot-swapping LoRA adapters on top of one shared frozen backbone.
  • On-device personalization: because LoRA adapters are tiny (megabytes, not gigabytes), they can be trained and stored on edge devices — enabling personalized models without cloud round-trips.
  • Research efficiency: academic labs that cannot afford A100 clusters can still publish competitive results by fine-tuning with LoRA on consumer hardware.

Beyond LoRA, the PEFT family includes QLoRA (quantize the base model to 4-bit, add LoRA on top — fits 65B models on a single 48 GB GPU), DoRA (weight-decomposed LoRA), and AdaLoRA (adaptive rank allocation). See also Neural Network Training for why gradient descent over billions of parameters is so expensive in the first place.

Conclusion

LoRA turns a simple linear-algebra observation — the rank of a matrix — into one of the most impactful engineering ideas in modern AI. A frozen pre-trained model carries most of the knowledge you need; adaptation only requires a thin low-dimensional nudge.

The result is striking: you can fine-tune a 7-billion-parameter language model on a laptop GPU by training fewer than 5 million parameters, swapping the adapter in and out instantly, and merging it back into the frozen weights at zero inference cost.

This is the broader lesson of PEFT: the hard part of intelligence is learned once. The easy part — adapting it to your task — barely touches the weights at all.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/lora-peft/Content licensed under CC BY-NC 4.0.