Model Quantization

A modern large language model stores its knowledge in billions of weights — floating-point numbers that tell every neuron how strongly to respond to every input. A 70-billion-parameter model in 32-bit float takes roughly 140 GB of memory. That fits in a data-center server rack, not a laptop or a phone.

Quantization is the art of replacing those heavy floats with lean integers. Map each weight from a 32-bit float to an 8-bit integer and the file shrinks to 35 GB. Drop to 4-bit and you get ~17.5 GB — suddenly a single consumer GPU can run a frontier model.

The idea sounds like simple rounding, but two things make it treacherous. First, a model's weights are not uniformly spread: a tiny fraction of neurons develop outlier activations whose values are 100× larger than the rest. Round those outliers carelessly and the model degrades catastrophically. Second, quantization errors compound layer by layer through a deep network, so even a small per-weight error can snowball into gibberish.

Solving those two problems is what separates naive quantization from production-grade methods like LLM.int8() (Dettmers et al., 2022) and GPTQ (Frantar et al., 2022), which together made it practical to run billion-parameter models on ordinary hardware.

Below is a small vector of model weights. Adjust the bit-width slider to quantize them — the bars show the original float values and the rounded integer approximations. The mean absolute error updates live.

<!-- {{c_html_intro}} -->
<div class="controls">
  <label>
    <span class="lbl">{{lbl_bits}}</span>
    <input type="range" id="bits" min="2" max="8" step="2" value="8">
    <span id="bits-val" class="badge">8</span>
  </label>
  <label class="toggle-row">
    <span class="lbl">{{lbl_outlier}}</span>
    <input type="checkbox" id="outlier" role="switch">
  </label>
</div>
<div class="chart" id="chart"></div>
<div class="stats" id="stats"></div>
<button id="resample" type="button">{{btn_resample}}</button>

/* {{c_css_intro}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
.controls { display: flex; flex-wrap: wrap; gap: .6rem 1.4rem; margin-bottom: .8rem; align-items: center; }
label { display: flex; align-items: center; gap: .4rem; font-size: .9rem; }
.lbl { font-weight: 600; }
input[type=range] { width: 120px; accent-color: #1d3557; }
.badge { background: #1d3557; color: #fff; border-radius: 4px; padding: 1px 6px; font-size: .85rem; min-width: 2ch; text-align: center; }
.toggle-row input[type=checkbox] { width: 36px; height: 20px; appearance: none; background: #ccc; border-radius: 10px; cursor: pointer; transition: background .2s; position: relative; }
.toggle-row input[type=checkbox]:checked { background: #e63946; }
.toggle-row input[type=checkbox]::after { content: ""; position: absolute; top: 2px; left: 2px; width: 16px; height: 16px; border-radius: 50%; background: #fff; transition: transform .2s; }
.toggle-row input[type=checkbox]:checked::after { transform: translateX(16px); }
/* {{c_css_chart}} */
.chart { display: flex; align-items: flex-end; gap: 3px; height: 160px; border-bottom: 2px solid #ccc; padding-bottom: 2px; overflow-x: auto; }
.bar-group { display: flex; flex-direction: column; align-items: center; gap: 1px; }
.bar-wrap { display: flex; gap: 2px; align-items: flex-end; height: 140px; }
.bar { width: 14px; border-radius: 3px 3px 0 0; min-height: 2px; transition: height .25s, background .25s; }
.bar.orig { background: #457b9d; }
.bar.quant { background: #e63946; opacity: .82; }
.bar-idx { font-size: .65rem; color: #888; margin-top: 1px; }
.stats { font-size: .9rem; margin: .5rem 0; min-height: 1.2em; }
.stats strong { color: #1d3557; }
button { font: 600 14px system-ui; padding: .4rem .85rem; border: 1px solid #1d3557; background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; margin-top: .3rem; }
.outlier-bar { background: #f4a261 !important; }
@media (prefers-color-scheme: dark) {
  body { color: #e0e0e0; }
  .bar.orig { background: #74b3ce; }
  .bar.quant { background: #ff6b6b; }
  .stats strong { color: #9fb2c8; }
  button { border-color: #74b3ce; background: #74b3ce; color: #111; }
}

// Code not found

Notice how the error stays low at 8-bit but jumps sharply at 4-bit and 2-bit. Now toggle the outlier switch: a single weight jumps to 10× the normal range. Because the quantization grid must now span a much wider range, every ordinary weight gets rounded more coarsely — and the error spikes across the whole vector. This is the outlier problem that LLM.int8() solves by handling outlier channels in float16 and only quantizing the rest.

Quantization looks like rounding, but the hard part is choosing which rounding minimizes the final output error.

• Naive round-to-nearest maps each weight independently to the closest integer level. It is fast and good enough for classical CNNs, but for large transformers the compound error can collapse accuracy on reasoning tasks even at int8.
• The outlier problem is why LLM.int8() (Dettmers et al., 2022) separates the computation: it detects which activation channels contain outliers (roughly 0.1% of channels in GPT-3-scale models) and keeps those in float16, quantizing the remaining 99.9% to int8. The mixed-precision matrix multiply is slower but preserves accuracy completely.
• GPTQ (Frantar et al., 2022) takes a different approach. It treats quantization as a layer-wise reconstruction problem: given the Hessian of the layer's output with respect to its weights, it updates the remaining unquantized weights to compensate for each rounding decision. The result is accurate int4 quantization — a 4× memory reduction over float16 — achieved in a few GPU-hours on a calibration dataset, without retraining.
• Why 4-bit beats 8-bit in practice: at scale, memory bandwidth is the bottleneck. A model that fits in GPU VRAM runs inference; one that doesn't, doesn't. GPTQ-int4 lets a 70B model run on two consumer GPUs instead of eight data-center GPUs, which is why it became the standard for open-weight model deployment.

The computational core of GPTQ is closely related to matrix multiplication and second-order optimization, and the outlier-detection strategy echoes ideas from dimensionality reduction.

The gap between a model's float32 size and a device's VRAM is where quantization lives:

• On-device inference: int4 and int8 models run entirely on smartphones and laptops via frameworks like llama.cpp and GGUF, enabling private, offline assistants with no cloud dependency.
• Single-GPU deployment: quantization is what lets a research lab or solo developer serve a 70B-class model on one or two consumer GPUs instead of a multi-GPU cluster.
• Speculative decoding and batching: smaller quantized models can serve as draft models, generating tokens cheaply before a larger model verifies them — a technique that multiplies throughput on production serving stacks.
• Embedded and IoT AI: 2-bit and 1-bit quantization (BitNet, 2024) pushes inference toward microcontrollers, where even kilobytes matter.

The tradeoff is always the same: lower bits mean less memory and faster math, but also a coarser grid and more rounding error. Modern methods spend their cleverness on minimizing that error rather than on reducing bits uniformly.

Quantization is one of the most quietly impactful ideas in modern AI engineering. It turns the theoretical question "how many bits does a weight really need?" into an optimization problem whose solution determines whether a frontier model runs on your laptop or only in a data center.

The outlier problem is a perfect illustration of how a tiny minority can dominate the whole: 0.1% of weight channels can force the other 99.9% to be quantized coarsely unless you treat them specially. GPTQ's answer — compensate for each rounding decision using second-order information — shows that even a problem as mundane as rounding can reward mathematical depth.

The next time you run a local LLM on a consumer GPU, remember: you are not running the original model. You are running a carefully optimized integer approximation of it — one that, against all expectation, reasons almost as well as the float original.