Columnar Compression

When a database stores a billion sales records, a traditional row store keeps every field for each sale together: date, product, region, amount, status — all bundled in one row. That layout is perfect for fetching a single customer's full record. But for analytics, where you want to sum the amount column across all rows, you end up reading every field just to get to the one you care about.

Column stores flip the layout. Every value of product lives next to every other product value; every status sits next to every other status. Suddenly a query that touches only two columns reads only two columns' worth of disk blocks, ignoring the rest.

The deeper gain is compression. When you read a column of sales statuses, you find the same handful of values — "confirmed", "pending", "refunded" — repeated millions of times. Row stores mix all columns together and lose that regularity; column stores expose it, and then squeeze it hard.

The three workhorses that do the squeezing are run-length encoding (RLE), dictionary encoding, and bit-packing — three elegant ideas that together can shrink a column to a fraction of its original size, sometimes 10× or more.

Type or edit the column values below — one value per line — then click Compress to watch all three encodings go to work. The size bars show how many bytes each encoding needs relative to the raw column.

<!-- {{c_html_intro}} -->
<p class="hint">{{hint_para}}</p>
<div class="editor-row">
  <div class="editor-col">
    <label class="col-label" for="col-input">{{label_column}}</label>
    <textarea id="col-input" rows="10" spellcheck="false"></textarea>
  </div>
  <div class="output-col" id="output-col">
    <p class="placeholder-msg">{{placeholder_msg}}</p>
  </div>
</div>
<div class="btns">
  <button id="btn-compress" type="button">{{btn_compress}}</button>
  <button id="btn-reset" type="button" class="ghost">{{btn_reset}}</button>
</div>
<div id="status" class="status"></div>

/* {{c_css_intro}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; padding: 14px; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .8rem; line-height: 1.5; }
.editor-row { display: flex; gap: .8rem; align-items: flex-start; flex-wrap: wrap; }
.editor-col { display: flex; flex-direction: column; min-width: 130px; flex: 0 0 130px; }
.output-col { flex: 1 1 200px; min-width: 200px; }
.col-label { font-size: .78rem; font-weight: 700; color: #555; margin-bottom: .3rem; }
textarea { font: 13px ui-monospace, monospace; padding: .4rem .5rem; border: 1px solid #bcc; border-radius: 6px;
           resize: vertical; width: 100%; line-height: 1.55; }
.placeholder-msg { font-size: .85rem; color: #888; font-style: italic; margin: 0; padding-top: .4rem; }
/* {{c_encoding_card}} */
.enc-card { border: 1px solid #d8e2ec; border-radius: 8px; padding: .6rem .8rem; margin-bottom: .55rem; background: #f8fafc; }
.enc-title { font-size: .8rem; font-weight: 700; color: #1d3557; margin: 0 0 .3rem; }
.bar-row { display: flex; align-items: center; gap: .5rem; margin-bottom: .3rem; }
.bar-bg { flex: 1; background: #e1eaf2; border-radius: 4px; height: 10px; overflow: hidden; }
.bar-fill { height: 100%; border-radius: 4px; background: #457b9d; transition: width .35s; }
.bar-label { font-size: .75rem; color: #333; white-space: nowrap; min-width: 70px; text-align: right; }
.enc-detail { font-size: .75rem; color: #555; margin: 0; line-height: 1.5; }
/* {{c_status_styles}} */
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin-top: .6rem; }
button { font: 600 13px system-ui; padding: .4rem .85rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.status { font-size: .85rem; font-weight: 600; margin-top: .4rem; min-height: 1.2em; color: #0a7d33; }

// Code not found

Notice how RLE wins on sorted or near-sorted data (long runs collapse to a count), dictionary wins when the column has few distinct values regardless of order, and bit-packing wins once the dictionary is small enough that every value fits in just a few bits. Real engines apply all three in sequence.

Each encoding attacks a different kind of redundancy.

Run-length encoding (RLE) replaces a run of identical values with (value, count) pairs. If your status column reads "confirmed" ten million times in a row, RLE replaces those ten million bytes with two numbers. Sorting the table by the column before compressing maximizes run lengths — a key insight behind engines like Apache Parquet and Vertica.

Dictionary encoding assigns a small integer code to each distinct value and stores only the codes. A column with four possible statuses needs just 2 bits per row instead of 9+ bytes per string. The dictionary itself is tiny. Reading the column means reading the compact codes and doing a cheap lookup — often faster than reading the raw strings even without counting the compression benefit.

Bit-packing takes dictionary codes a step further. If the dictionary has at most $2^{k}$ entries, each code fits in exactly $k$ bits. A standard integer wastes $32 - k$ bits per value. Pack them tightly and you save those wasted bits across every row. For a column with 16 distinct values, $k = 4$: you store 8 values per 32-bit word instead of 1, an 8× density gain.

The encodings compose: sort → RLE → dictionary → bit-pack is a common pipeline. Each stage hands off a smaller, more regular representation to the next. The result is that queries on compressed columns often run faster than on uncompressed data, because reading fewer bytes from disk or cache dominates the tiny decompression cost. This is the counterintuitive heart of modern compression in databases.

Columnar compression is the silent engine behind most modern analytical systems:

• Data warehouses: Snowflake, BigQuery, Redshift and Vertica all store data column-by-column with layered encodings. A single query scanning billions of rows can finish in seconds precisely because most of the data never leaves disk.
• Open formats: Apache Parquet and ORC are the de-facto storage formats for data lakes. Both use dictionary encoding and bit-packing by default, making Spark and Dask queries dramatically faster than reading raw CSV.
• In-memory analytics: DuckDB and Apache Arrow keep columns in RAM with the same encodings, so even in-memory joins operate on compressed representations and stay cache-friendly.
• Time-series databases: sensor readings repeat values and follow predictable patterns; delta encoding (store differences between successive values) combined with RLE can compress timestamps and measurements by 100× or more.

Understanding columnar compression also clarifies why pattern matching and aggregation queries are so much faster in analytical engines than in transactional ones — the data format itself is optimized for reading one attribute at a time across many rows, not one row at a time.

Columnar compression is a beautiful example of a format choice that makes an entire class of computation easier. By keeping values of the same type together, column stores expose the repetition that real data always contains — and then squeeze it with three complementary encodings: RLE turns runs into counts, dictionaries replace strings with tiny codes, and bit-packing fits those codes into the minimum number of bits.

The result is a system where reading less data and decompressing it is faster than reading the raw data was in the first place. That counterintuitive win is why every major analytics engine converged on columnar storage, and why formats like Parquet have become the universal language of large-scale data processing.