Introduction

When a database stores a billion sales records, a traditional row store keeps every field for each sale together: date, product, region, amount, status — all bundled in one row. That layout is perfect for fetching a single customer's full record. But for analytics, where you want to sum the amount column across all rows, you end up reading every field just to get to the one you care about.

Column stores flip the layout. Every value of product lives next to every other product value; every status sits next to every other status. Suddenly a query that touches only two columns reads only two columns' worth of disk blocks, ignoring the rest.

The deeper gain is compression. When you read a column of sales statuses, you find the same handful of values — "confirmed", "pending", "refunded" — repeated millions of times. Row stores mix all columns together and lose that regularity; column stores expose it, and then squeeze it hard.

The three workhorses that do the squeezing are run-length encoding (RLE), dictionary encoding, and bit-packing — three elegant ideas that together can shrink a column to a fraction of its original size, sometimes 10× or more.

Watch a Column Shrink

Type or edit the column values below — one value per line — then click Compress to watch all three encodings go to work. The size bars show how many bytes each encoding needs relative to the raw column.

<!-- Columnar compression demo — textarea column input + compression output cards -->
<p class="hint">Each line is one cell in a database column. Click <b>Compress</b> to apply run-length encoding (RLE), dictionary encoding, and bit-packing — and see how many bytes each method needs.</p>
<div class="editor-row">
  <div class="editor-col">
    <label class="col-label" for="col-input">Column values</label>
    <textarea id="col-input" rows="10" spellcheck="false"></textarea>
  </div>
  <div class="output-col" id="output-col">
    <p class="placeholder-msg">Compression results will appear here.</p>
  </div>
</div>
<div class="btns">
  <button id="btn-compress" type="button">Compress</button>
  <button id="btn-reset" type="button" class="ghost">Reset</button>
</div>
<div id="status" class="status"></div>

/* Layout: editor textarea on the left, compression result cards on the right */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; padding: 14px; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .8rem; line-height: 1.5; }
.editor-row { display: flex; gap: .8rem; align-items: flex-start; flex-wrap: wrap; }
.editor-col { display: flex; flex-direction: column; min-width: 130px; flex: 0 0 130px; }
.output-col { flex: 1 1 200px; min-width: 200px; }
.col-label { font-size: .78rem; font-weight: 700; color: #555; margin-bottom: .3rem; }
textarea { font: 13px ui-monospace, monospace; padding: .4rem .5rem; border: 1px solid #bcc; border-radius: 6px;
           resize: vertical; width: 100%; line-height: 1.55; }
.placeholder-msg { font-size: .85rem; color: #888; font-style: italic; margin: 0; padding-top: .4rem; }
/* encoding result card styles */
.enc-card { border: 1px solid #d8e2ec; border-radius: 8px; padding: .6rem .8rem; margin-bottom: .55rem; background: #f8fafc; }
.enc-title { font-size: .8rem; font-weight: 700; color: #1d3557; margin: 0 0 .3rem; }
.bar-row { display: flex; align-items: center; gap: .5rem; margin-bottom: .3rem; }
.bar-bg { flex: 1; background: #e1eaf2; border-radius: 4px; height: 10px; overflow: hidden; }
.bar-fill { height: 100%; border-radius: 4px; background: #457b9d; transition: width .35s; }
.bar-label { font-size: .75rem; color: #333; white-space: nowrap; min-width: 70px; text-align: right; }
.enc-detail { font-size: .75rem; color: #555; margin: 0; line-height: 1.5; }
/* button and status styles */
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin-top: .6rem; }
button { font: 600 13px system-ui; padding: .4rem .85rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.status { font-size: .85rem; font-weight: 600; margin-top: .4rem; min-height: 1.2em; color: #0a7d33; }

/* Columnar compression demo: RLE, dictionary encoding, bit-packing */
const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));

/* default column values loaded on startup and reset */
const DEFAULT_COLUMN = [
  'confirmed','confirmed','confirmed','pending','pending',
  'refunded','confirmed','confirmed','confirmed','confirmed'
];

/* Encoding helper functions */

/* Run-length encoding: collapse runs of identical values into (value, count) pairs */
function encodeRLE(values) {
  if (!values.length) return [];
  const runs = [];
  let cur = values[0], count = 1;
  for (let i = 1; i < values.length; i++) {
    if (values[i] === cur) { count++; }
    else { runs.push([cur, count]); cur = values[i]; count = 1; }
  }
  runs.push([cur, count]);
  return runs;
}

/* Byte cost of RLE: value string + separator + 1-byte count (or 4 bytes for large counts) */
function rleByteCost(runs) {
  return runs.reduce((s, [v, c]) => s + v.length + 1 + (c < 256 ? 1 : 4), 0);
}

/* Dictionary encoding: assign integer codes to distinct values */
function buildDict(values) {
  const seen = []; const map = {};
  for (const v of values) { if (!(v in map)) { map[v] = seen.length; seen.push(v); } }
  return { dict: seen, map };
}

/* Bits needed to represent n distinct values */
function bitsNeeded(n) { return n <= 1 ? 1 : Math.ceil(Math.log2(n)); }

/* Raw byte cost: each value string + 1 newline separator */
function rawBytes(values) { return values.reduce((s, v) => s + v.length + 1, 0); }

function compress() {
  const raw = document.getElementById('col-input').value
    .split('\n').map(function(s){ return s.trim(); }).filter(Boolean);
  if (!raw.length) { setStatus('Enter at least one value.'); return; }

  const rawB = rawBytes(raw);

  /* Step 1: Run-length encode */
  const runs = encodeRLE(raw);
  const rleB = rleByteCost(runs);
  const rlePct = Math.round(rleB / rawB * 100);

  /* Step 2: Dictionary encode (also computes bit-width k) */
  const dictResult = buildDict(raw);
  const dict = dictResult.dict;
  const k = bitsNeeded(dict.length);
  const dictHeaderB = dict.reduce(function(s, v){ return s + v.length + 1; }, 0);
  const dictCodesB = Math.ceil(raw.length * k / 8);
  const dictTotalB = dictHeaderB + dictCodesB;
  const dictPct = Math.round(dictTotalB / rawB * 100);

  /* Step 3: Bit-packing uses k bits per value (same k as dictionary) */
  const bitpackB = Math.ceil(raw.length * k / 8);
  const bitpackPct = Math.round(bitpackB / rawB * 100);

  /* Build output HTML cards */
  const rleDetail = i18nFmt('%runs% run(s) from %raw% row(s) — (value, count) pairs.', { runs: runs.length, raw: raw.length });
  const dictDetail = i18nFmt('%distinct% distinct value(s) → %k%-bit codes × %rows% row(s), plus the dictionary.', { distinct: dict.length, k: k, rows: raw.length });
  const bitDetail = i18nFmt('%k% bit(s) per value → %per_word% values per 32-bit word.', { k: k, per_word: Math.floor(32 / k) });

  document.getElementById('output-col').innerHTML =
    '<div class="enc-card">' +
      '<p class="enc-title">Raw (uncompressed)</p>' +
      '<div class="bar-row">' +
        '<div class="bar-bg"><div class="bar-fill" style="width:100%;background:#888"></div></div>' +
        '<span class="bar-label">' + rawB + ' bytes</span>' +
      '</div>' +
      '<p class="enc-detail">One byte per character plus a separator per row — no compression.</p>' +
    '</div>' +
    '<div class="enc-card">' +
      '<p class="enc-title">Run-Length Encoding (RLE)</p>' +
      '<div class="bar-row">' +
        '<div class="bar-bg"><div class="bar-fill" style="width:' + Math.min(rlePct,100) + '%"></div></div>' +
        '<span class="bar-label">' + rleB + ' bytes (' + rlePct + '%)</span>' +
      '</div>' +
      '<p class="enc-detail">' + rleDetail + '</p>' +
    '</div>' +
    '<div class="enc-card">' +
      '<p class="enc-title">Dictionary Encoding</p>' +
      '<div class="bar-row">' +
        '<div class="bar-bg"><div class="bar-fill" style="width:' + Math.min(dictPct,100) + '%"></div></div>' +
        '<span class="bar-label">' + dictTotalB + ' bytes (' + dictPct + '%)</span>' +
      '</div>' +
      '<p class="enc-detail">' + dictDetail + '</p>' +
    '</div>' +
    '<div class="enc-card">' +
      '<p class="enc-title">Bit-Packing (after dictionary)</p>' +
      '<div class="bar-row">' +
        '<div class="bar-bg"><div class="bar-fill" style="width:' + Math.min(bitpackPct,100) + '%;background:#e63946"></div></div>' +
        '<span class="bar-label">' + bitpackB + ' bytes (' + bitpackPct + '%)</span>' +
      '</div>' +
      '<p class="enc-detail">' + bitDetail + '</p>' +
    '</div>';

  setStatus(i18nFmt('Raw: %raw% bytes → best encoding: %best% bytes.', { raw: rawB, best: Math.min(rleB, dictTotalB, bitpackB) }));
}

function resetDemo() {
  document.getElementById('col-input').value = DEFAULT_COLUMN.join('\n');
  document.getElementById('output-col').innerHTML = '<p class="placeholder-msg">Compression results will appear here.</p>';
  setStatus('');
}

function setStatus(t) {
  document.getElementById('status').textContent = t;
}

document.getElementById('btn-compress').addEventListener('click', compress);
document.getElementById('btn-reset').addEventListener('click', resetDemo);
resetDemo();
Result
Console

Notice how RLE wins on sorted or near-sorted data (long runs collapse to a count), dictionary wins when the column has few distinct values regardless of order, and bit-packing wins once the dictionary is small enough that every value fits in just a few bits. Real engines apply all three in sequence.

The Real Complexity

Each encoding attacks a different kind of redundancy.

Run-length encoding (RLE) replaces a run of identical values with (value, count) pairs. If your status column reads "confirmed" ten million times in a row, RLE replaces those ten million bytes with two numbers. Sorting the table by the column before compressing maximizes run lengths — a key insight behind engines like Apache Parquet and Vertica.

Dictionary encoding assigns a small integer code to each distinct value and stores only the codes. A column with four possible statuses needs just 2 bits per row instead of 9+ bytes per string. The dictionary itself is tiny. Reading the column means reading the compact codes and doing a cheap lookup — often faster than reading the raw strings even without counting the compression benefit.

Bit-packing takes dictionary codes a step further. If the dictionary has at most 2k2^{k} entries, each code fits in exactly kk bits. A standard integer wastes 32−k32 - k bits per value. Pack them tightly and you save those wasted bits across every row. For a column with 16 distinct values, k=4k = 4: you store 8 values per 32-bit word instead of 1, an 8× density gain.

The encodings compose: sort → RLE → dictionary → bit-pack is a common pipeline. Each stage hands off a smaller, more regular representation to the next. The result is that queries on compressed columns often run faster than on uncompressed data, because reading fewer bytes from disk or cache dominates the tiny decompression cost. This is the counterintuitive heart of modern compression in databases.

Where It Matters

Columnar compression is the silent engine behind most modern analytical systems:

  • Data warehouses: Snowflake, BigQuery, Redshift and Vertica all store data column-by-column with layered encodings. A single query scanning billions of rows can finish in seconds precisely because most of the data never leaves disk.
  • Open formats: Apache Parquet and ORC are the de-facto storage formats for data lakes. Both use dictionary encoding and bit-packing by default, making Spark and Dask queries dramatically faster than reading raw CSV.
  • In-memory analytics: DuckDB and Apache Arrow keep columns in RAM with the same encodings, so even in-memory joins operate on compressed representations and stay cache-friendly.
  • Time-series databases: sensor readings repeat values and follow predictable patterns; delta encoding (store differences between successive values) combined with RLE can compress timestamps and measurements by 100× or more.

Understanding columnar compression also clarifies why pattern matching and aggregation queries are so much faster in analytical engines than in transactional ones — the data format itself is optimized for reading one attribute at a time across many rows, not one row at a time.

Conclusion

Columnar compression is a beautiful example of a format choice that makes an entire class of computation easier. By keeping values of the same type together, column stores expose the repetition that real data always contains — and then squeeze it with three complementary encodings: RLE turns runs into counts, dictionaries replace strings with tiny codes, and bit-packing fits those codes into the minimum number of bits.

The result is a system where reading less data and decompressing it is faster than reading the raw data was in the first place. That counterintuitive win is why every major analytics engine converged on columnar storage, and why formats like Parquet have become the universal language of large-scale data processing.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/columnar-compression/Content licensed under CC BY-NC 4.0.