LoRA & Parameter-Efficient Fine-Tuning

Training a model like GPT-3 from scratch costs tens of millions of dollars. But what if you just want to teach the same model to write legal summaries, answer medical questions, or speak a new language? Full fine-tuning reruns the whole optimization over billions of weights — most of which barely need to move at all.

LoRA (Low-Rank Adaptation, Hu et al., 2021) makes a simple but powerful observation: the change in a weight matrix during fine-tuning lies on a very low-dimensional surface. Instead of updating the full matrix $W \in \mathbb{R}^{d \times k}$, LoRA freezes $W$ and injects two small matrices $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ (where $r \ll d, k$). Only $B$ and $A$ are trained; the effective update is $\Delta W = BA$.

With a rank as low as $r = 4$, LoRA reduces the trainable parameters for a large transformer layer from tens of millions to a few thousand — yet matches full fine-tuning on most benchmarks. The frozen weights cost no gradient memory, and the adapters can be swapped instantly between tasks.

LoRA belongs to a family called PEFT (Parameter-Efficient Fine-Tuning), which also includes prompt tuning, prefix tuning, and adapter layers. All share the same goal: adapt a frozen giant with a tiny trainable patch.

A weight matrix $W$ is fixed (frozen). LoRA trains only two small matrices $B$ and $A$ whose product $\Delta W = BA$ approximates the desired change. Drag the rank slider to see how many parameters you actually train versus the full matrix, and watch the low-rank approximation fill in.

<!-- {{c_html_desc}} -->
<div class="demo-wrap">
  <div class="controls">
    <label for="rank-slider">{{lbl_rank}} <strong id="rank-val">4</strong>
      &nbsp;<span class="dim-label">(d=<span id="lbl-d">8</span>, k=<span id="lbl-k">8</span>)</span>
    </label>
    <input id="rank-slider" type="range" min="1" max="8" value="4" step="1">
  </div>
  <div class="param-row">
    <div class="param-box full-box">
      <div class="param-title">{{lbl_full_params}}</div>
      <div class="param-num" id="full-params">64</div>
      <div class="param-sub">d × k</div>
    </div>
    <div class="param-box lora-box">
      <div class="param-title">{{lbl_lora_params}}</div>
      <div class="param-num" id="lora-params">64</div>
      <div class="param-sub">r(d + k)</div>
    </div>
    <div class="param-box save-box">
      <div class="param-title">{{lbl_reduction}}</div>
      <div class="param-num" id="reduction">1×</div>
      <div class="param-sub">{{lbl_fewer}}</div>
    </div>
  </div>
  <div class="matrix-section">
    <div class="mat-wrap">
      <div class="mat-title">W ({{lbl_frozen}})</div>
      <canvas id="canvas-W" width="128" height="128"></canvas>
    </div>
    <div class="mat-wrap">
      <div class="mat-title">ΔW = B·A ({{lbl_trained}})</div>
      <canvas id="canvas-DW" width="128" height="128"></canvas>
    </div>
    <div class="mat-wrap">
      <div class="mat-title">W + ΔW ({{lbl_adapted}})</div>
      <canvas id="canvas-WP" width="128" height="128"></canvas>
    </div>
  </div>
  <div class="error-row">
    <span>{{lbl_approx_error}}: </span><strong id="approx-err">—</strong>
    &nbsp;|&nbsp;
    <span>{{lbl_rank_label}}: </span><strong id="rank-label">—</strong>
  </div>
  <div class="legend">
    <span class="leg-neg">{{leg_neg}}</span>
    <span class="leg-zero">0</span>
    <span class="leg-pos">{{leg_pos}}</span>
  </div>
</div>

/* {{c_css_desc}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; background: transparent; }
.demo-wrap { padding: .6rem .4rem; max-width: 520px; margin: 0 auto; }
.controls { margin-bottom: .6rem; }
label { font-size: .9rem; color: #444; display: block; margin-bottom: .3rem; }
.dim-label { font-size: .8rem; color: #777; }
input[type=range] { width: 100%; accent-color: #1d3557; }
.param-row { display: flex; gap: .5rem; margin-bottom: .8rem; }
.param-box { flex: 1; border-radius: 8px; padding: .5rem .4rem; text-align: center; border: 1px solid #cdd9e3; }
.full-box { background: #e8eef3; }
.lora-box { background: #d4edda; }
.save-box { background: #fff3cd; }
.param-title { font-size: .72rem; color: #555; margin-bottom: .15rem; }
.param-num { font-size: 1.3rem; font-weight: 700; color: #1d3557; }
.param-sub { font-size: .68rem; color: #888; }
.matrix-section { display: flex; gap: .6rem; justify-content: center; margin-bottom: .5rem; flex-wrap: wrap; }
.mat-wrap { text-align: center; }
.mat-title { font-size: .75rem; color: #444; margin-bottom: .2rem; }
canvas { border: 1px solid #cdd9e3; border-radius: 4px; image-rendering: pixelated; display: block; }
.error-row { font-size: .82rem; color: #444; margin-bottom: .4rem; }
.legend { display: flex; gap: .5rem; font-size: .75rem; align-items: center; }
.leg-neg { background: #c92f3c; color: #fff; padding: 1px 6px; border-radius: 3px; }
.leg-zero { background: #e8eef3; padding: 1px 6px; border-radius: 3px; }
.leg-pos { background: #0a7d33; color: #fff; padding: 1px 6px; border-radius: 3px; }

// Code not found

Notice the trade-off: at rank 1 you train almost nothing but the approximation is coarse; as rank approaches the full dimension $d$ you recover full fine-tuning at full cost. In practice, $r \in \{4, 8, 16\}$ captures most of what a task needs while keeping the parameter count tiny.

Why does a rank-4 update capture most of what fine-tuning needs? The answer is the intrinsic dimensionality hypothesis.

Aghajanyan et al. (2021) showed empirically that pre-trained language models can be fine-tuned by optimizing in a random low-dimensional subspace of the full parameter space. The model has learned such a rich general representation that adapting it to a specific task only requires movement along a handful of directions — not a full re-sculpting of all billions of weights.

LoRA makes this precise for a single weight matrix:

• Full update: $W' = W + \Delta W$, train $d \times k$ parameters.
• LoRA update: $\Delta W = BA$, train only $r(d + k)$ parameters, where $r \ll \min(d, k)$.
• At inference time: merge $W' = W + BA$ (no runtime overhead) or keep $B, A$ separate for instant task-switching.

The rank $r$ is a hyperparameter that controls the expressivity-efficiency trade-off. For very large models ($d = 4096$, $k = 4096$), even $r = 8$ trains $8 \times 8192 = 65\,536$ parameters versus $16\,777\,216$ for the full matrix — a 256× reduction.

The initialization matters too: $B$ starts at zero (so $\Delta W = BA = 0$ at step 0, identical to the pre-trained model), and $A$ is drawn from a Gaussian. This guarantees the adapted model starts exactly where the pre-trained model left off.

LoRA is now the backbone of most open-source fine-tuning pipelines for models like LLaMA, Mistral, and GPT-style architectures. See also how Transformers & Attention create the weight matrices that LoRA adapts.

LoRA and PEFT techniques power nearly every practical large-model deployment today:

• Instruction tuning: teaching a base model to follow chat instructions (ChatGPT-style RLHF pipelines use LoRA for the SFT step) without touching the billions of base weights.
• Domain adaptation: a single base model plus a 10 MB LoRA adapter becomes a legal-document analyst, a medical-record coder, or a customer-service bot — all without separate full-size copies.
• Multi-task serving: a single GPU can serve dozens of tasks by hot-swapping LoRA adapters on top of one shared frozen backbone.
• On-device personalization: because LoRA adapters are tiny (megabytes, not gigabytes), they can be trained and stored on edge devices — enabling personalized models without cloud round-trips.
• Research efficiency: academic labs that cannot afford A100 clusters can still publish competitive results by fine-tuning with LoRA on consumer hardware.

Beyond LoRA, the PEFT family includes QLoRA (quantize the base model to 4-bit, add LoRA on top — fits 65B models on a single 48 GB GPU), DoRA (weight-decomposed LoRA), and AdaLoRA (adaptive rank allocation). See also Neural Network Training for why gradient descent over billions of parameters is so expensive in the first place.

LoRA turns a simple linear-algebra observation — the rank of a matrix — into one of the most impactful engineering ideas in modern AI. A frozen pre-trained model carries most of the knowledge you need; adaptation only requires a thin low-dimensional nudge.

The result is striking: you can fine-tune a 7-billion-parameter language model on a laptop GPU by training fewer than 5 million parameters, swapping the adapter in and out instantly, and merging it back into the frozen weights at zero inference cost.

This is the broader lesson of PEFT: the hard part of intelligence is learned once. The easy part — adapting it to your task — barely touches the weights at all.