JPEG and the Cosine Transform

Every time you share a photo, open a webpage, or video-call someone, a piece of mathematics called the Discrete Cosine Transform is silently at work. JPEG — the format behind most photos on the internet — uses it to shrink a raw image by a factor of ten or more while keeping the result indistinguishable from the original to the human eye.

The core idea is disarmingly simple: not all image information is equally visible. The fine-grained, rapidly oscillating patterns — the high-frequency detail — are things our eyes struggle to see at normal viewing distances. Throw those away, and you barely notice. Keep the broad strokes — the low-frequency shapes and gentle gradients — and the image still looks sharp.

JPEG was standardized in 1992 by the Joint Photographic Experts Group, and the algorithm at its heart has not changed. Understanding it means understanding one of the most successful ideas in all of applied mathematics: transform a problem into a space where you can separate the important from the expendable, then discard the rest.

The slider below controls how many DCT coefficients are kept from each 8×8 block — low numbers keep only the lowest-frequency information, high numbers keep almost everything. Watch how image quality and the retained data change.

<div class="controls">
  <label for="quality">{{kept_coeffs}} <strong id="qualLabel">10</strong> {{of_64}}</label>
  <input type="range" id="quality" min="1" max="64" value="10">
  <span class="size-note" id="sizeNote"></span>
</div>
<div class="canvases">
  <div class="canvas-wrap">
    <div class="caption">{{caption_original}}</div>
    <canvas id="origCanvas" width="128" height="128"></canvas>
  </div>
  <div class="canvas-wrap">
    <div class="caption">{{caption_dct_prefix}} <span id="keepLabel">10</span> {{caption_dct_suffix}}</div>
    <canvas id="dctCanvas" width="128" height="128"></canvas>
  </div>
  <div class="canvas-wrap">
    <div class="caption">{{caption_coeffs}}</div>
    <canvas id="coeffCanvas" width="128" height="128"></canvas>
  </div>
</div>
<div class="info" id="info"></div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
.controls { display: flex; flex-direction: column; gap: .4rem; margin-bottom: .8rem; }
label { font-size: .9rem; color: #333; }
input[type=range] { width: 100%; max-width: 340px; accent-color: #1d3557; }
.size-note { font-size: .8rem; color: #556; }
.canvases { display: flex; gap: 10px; flex-wrap: wrap; margin-bottom: .6rem; }
.canvas-wrap { display: flex; flex-direction: column; align-items: center; gap: 4px; }
.caption { font-size: .78rem; color: #445; text-align: center; max-width: 128px; }
canvas { border: 1px solid #cdd9e3; border-radius: 6px; image-rendering: pixelated; }
.info { font-size: .85rem; color: #1d3557; min-height: 1.4em; }

// Code not found

Notice that even at very low quality the broad shapes survive. The high-frequency coefficients — which encode sharp edges and fine textures — are the first to go. At mid quality the image already looks acceptable; the data savings are enormous. This is the essential trade-off JPEG makes: bet on what human vision cannot resolve, and be right almost every time.

Beneath the slider is a precise algorithm running in well-understood time.

• The DCT formula. For an 8×8 block of pixel values, the DCT computes 64 coefficients. Coefficient (u, v) captures how much of the cosine wave with horizontal frequency u and vertical frequency v is present in the block. Low (u, v) = slow oscillation = broad shapes. High (u, v) = fast oscillation = fine texture.
• Complexity. A naive DCT on an n-point signal is $O(n^{2})$. With the Fast Fourier Transform relationship, it can be reduced to $O(n \log n)$ — the same family as sorting. For the fixed 8×8 block, this is just 64 constants regardless of image size.
• Quantization. After the DCT, each coefficient is divided by a number from a quantization table and rounded to the nearest integer. This is where information is actually thrown away. High-frequency slots get large divisors (coarse rounding = heavy loss). Low-frequency slots get small divisors (fine rounding = light loss). The quality setting you choose adjusts these divisors.
• Entropy coding. The quantized values — mostly zeros at low quality — are then losslessly compressed with Huffman codes, squeezing out the remaining redundancy.
• Why it works. Natural images have most of their energy in low frequencies. The DCT "compacts" this energy into the first few coefficients — an empirical fact known as energy compaction — which is why aggressive truncation of the rest is nearly invisible.

The whole standard is proven to be optimal among transform coders for natural images with Gaussian statistics. It is one of the few cases where the theoretically best answer and the practically deployed one are the same thing.

The DCT is one of the most deployed algorithms in human history:

• Digital photography. Every camera's "JPEG quality" setting is controlling the quantization table. The majority of photos ever taken are stored using this algorithm.
• Video compression. MPEG-1, MPEG-2 (DVD), H.264 (streaming, Blu-ray), and H.265 all use DCT-based transforms on macroblocks. The video call you are on right now almost certainly uses it.
• Audio compression. MP3 and AAC apply a closely related Modified DCT (MDCT) to audio frames — the same principle: transform, quantize the inaudible, compress.
• Medical imaging. DICOM, the format for CT and MRI scans, can use JPEG compression for preview images and progressive transmission.
• WebP and HEIC. Newer formats use more sophisticated transforms (HEIC uses HEVC's integer DCT) but the conceptual lineage goes straight back to JPEG's 8×8 block idea.

The transform also appears in dimensionality reduction and signal processing far beyond images: any time you want to represent a signal compactly, the frequency domain is the first place to look.

JPEG is a beautiful example of mathematics meeting biology. The Discrete Cosine Transform does not know about eyes — it only computes frequencies. But by pairing it with a quantization table tuned to human perception, engineers in 1992 created an algorithm that is simultaneously theoretically optimal and practically invisible.

The key insight — transform first, then throw away what cannot be seen — reappears in every modern codec, every audio format, and every image standard that followed. The numbers change; the principle does not.

The next time you share a photo and marvel that it fits in a few hundred kilobytes, remember: somewhere in that file are 64 cosine waves per block, with most of their amplitudes quietly set to zero. You don't see the loss, and that is the entire point. For related ideas, explore dimensionality reduction or sorting lower bounds.