Shannon Entropy

How much information is in a message? It feels like a vague, almost philosophical question — until you try to send it down a wire or store it on a disk and need an exact answer in bits.

In 1948, Claude Shannon gave one. In a single paper, A Mathematical Theory of Communication, he defined entropy: a precise measure, in bits, of how much information a source produces. The idea is simple and surprising — information is surprise. A symbol you fully expect carries almost nothing; a symbol you didn't see coming carries a lot.

From that one definition follows a hard, unbreakable rule: there is a floor on how small any lossless code can make your message, and that floor is the entropy. No compressor — not ZIP, not anything ever invented — can beat it.

Type any message below. The demo counts how often each symbol appears, turns those frequencies into probabilities, and computes the entropy — the average number of bits each symbol carries. Multiply by the message length and you get the smallest size, in bits, that any lossless code could ever reach.

<p class="hint">{{hint}}</p>
<input id="msg" type="text" value="{{initial_msg}}" autocomplete="off" spellcheck="false" />
<div class="btns">
  <button id="ex-flat" type="button">{{btn_flat}}</button>
  <button id="ex-mix" type="button">{{btn_mix}}</button>
  <button id="ex-eng" type="button">{{btn_eng}}</button>
</div>
<div class="readout">
  <div class="stat"><span class="num" id="H">0.00</span><span class="lbl">{{lbl_entropy}}</span></div>
  <div class="stat"><span class="num" id="floor">0</span><span class="lbl">{{lbl_min}}</span></div>
  <div class="stat"><span class="num" id="naive">0</span><span class="lbl">{{lbl_fixed}}</span></div>
</div>
<div class="bars" id="bars"></div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
#msg { width: 100%; font: 600 16px ui-monospace, monospace; padding: .5rem .6rem;
       border: 1px solid #cdd9e3; border-radius: 8px; margin-bottom: .5rem; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin-bottom: .8rem; }
button { font: 600 13px system-ui, sans-serif; padding: .4rem .8rem; border: 1px solid #1d3557;
         background: #fff; color: #1d3557; border-radius: 8px; cursor: pointer; }
button:hover { background: #eef3f8; }
.readout { display: flex; gap: .8rem; flex-wrap: wrap; margin-bottom: .9rem; }
.stat { background: #e8eef3; border: 1px solid #cdd9e3; border-radius: 10px;
        padding: .55rem .8rem; min-width: 120px; flex: 1; }
.num { display: block; font: 800 22px ui-monospace, monospace; color: #1d3557; }
.lbl { font-size: .72rem; color: #51606e; line-height: 1.25; display: block; margin-top: .15rem; }
.bars { display: flex; flex-direction: column; gap: 4px; }
.row { display: flex; align-items: center; gap: 8px; font: 600 13px ui-monospace, monospace; }
.row .sym { width: 26px; text-align: center; background: #f0f3f6; border-radius: 5px; padding: 2px 0; }
.row .bar { height: 14px; background: #1d3557; border-radius: 4px; min-width: 2px; }
.row .pct { font-size: .72rem; color: #51606e; }

// Code not found

Watch the extremes. A message of all the same character (aaaaaa) has zero entropy — once you know the alphabet, every symbol is fully predictable, so it carries no information at all. Mix in more distinct, equally likely symbols and the entropy climbs toward its maximum, $\log_{2}$ of the alphabet size. That number is a wall: pack below it and you would lose information that can never be recovered.

What status does this limit have? It is not a guess or a rule of thumb — it is a proven theorem.

• Entropy is defined as $H = -\sum_p p \log_2 p$, summed over every symbol with probability $p$. It is the average surprise per symbol, in bits.
• Source coding theorem (Shannon, 1948). Any lossless code needs at least H bits per symbol on average. You can get arbitrarily close to H with clever coding (Huffman, arithmetic coding), but you can never go below it. This is settled mathematics, not an open question.
• Channel capacity (Shannon, 1948). A noisy channel has a maximum rate C at which information can be sent with vanishingly small error. Push past C and reliable communication becomes impossible — another hard wall.
• A subtle cousin. Entropy bounds compression on average over a known source. The information in a single specific string is its Kolmogorov complexity — related, but uncomputable in general. Entropy stays computable and practical.

So unlike the famously open problems of computer science, this one is closed. Shannon didn't just propose a measure; he proved it is the true boundary. The floor is real, and it has a number.

Entropy is one of those rare ideas that quietly runs under almost everything digital:

• Compression. ZIP, PNG, MP3 and H.264 all chase the entropy floor. Knowing H tells engineers exactly how much room is left to squeeze — and when to stop trying.
• Reliable communication. Wi-Fi, 5G, deep-space probes and hard drives use error-correcting codes designed right up against the channel-capacity limit.
• Security. "Password entropy" measures unpredictability in bits; more entropy means exponentially more guesses to crack.
• Machine learning. Cross-entropy is the standard loss for classifiers — training a model is literally minimizing the bits needed to describe the right answers.

The unifying picture connects straight to data compression and the limits of computing π: once you know the information content, you know the budget, and physics won't let you cheat it.

Shannon gave the fuzzy word information an exact size. Count how often each symbol shows up, feed the probabilities into one formula, and out comes the number of bits a message truly carries — and the smallest it could ever be made without losing anything.

That floor is proven, not hoped for. So when your compressor stalls and refuses to make a file any smaller, it isn't failing — it has hit the entropy. You are standing at a boundary drawn in 1948, the same one that shapes data compression and every channel that carries our data, and there is simply no path below it.