Introduction

Every function call in a C-style program uses the stack: a strip of memory that grows downward. When foo() calls bar(), the CPU pushes a return address — the instruction to jump back to when bar finishes — right next to the local variables of bar.

Most of those local variables are buffers: fixed-size byte arrays meant to hold input. The classic C functions gets, strcpy, and scanf copy bytes into a buffer without checking its size. Write more bytes than the buffer holds and they spill into adjacent memory — first into other local variables, and then, crucially, into the saved return address.

Overwrite that address with a value the attacker controls, and the CPU will "return" to whatever instruction they choose. That is a buffer overflow exploit: a bug so reliable and so severe that it shaped the entire history of systems security.

The mechanism was known by the 1970s, but the landmark paper by Aleph One, Smashing the Stack for Fun and Profit (Phrack, 1996), made it accessible to a generation of security researchers and attackers alike. The story since then is a race between increasingly clever exploits and increasingly robust defenses.

Try It: Overflow the Buffer

The demo below shows a simplified stack frame with a 4-byte buffer, a stack canary, and a saved return address. Type characters in the input field and watch the bytes fill the buffer — and then spill beyond it.

<!-- Educational buffer overflow stack frame demo -->
<div class="demo-wrap">
  <p class="hint">Type into the box — each character is one byte. The stack frame holds a <b>4-byte buffer</b>, a <b>stack canary</b>, and the <b>saved return address</b>. Writing past byte 4 overwrites the canary; past byte 5 corrupts the return address.</p>
  <div class="input-row">
    <input id="user-input" type="text" maxlength="16" placeholder="Type here to fill the buffer…" autocomplete="off" spellcheck="false">
    <button id="btn-reset" type="button" class="ghost">Reset</button>
  </div>
  <!-- Stack frame visualization — rows from low (bottom of stack) to high address -->
  <div class="stack-diagram" id="stack-diagram" aria-label="Stack frame diagram">
    <div class="row header-row">
      <span class="label">Address</span>
      <span class="label">Value</span>
      <span class="label">Slot</span>
    </div>
    <div class="row" id="row-ret-hi">
      <span class="addr">0x18</span>
      <span class="cell" id="cell-ret-hi"></span>
      <span class="cell-label">return address (high)</span>
    </div>
    <div class="row" id="row-ret-lo">
      <span class="addr">0x14</span>
      <span class="cell" id="cell-ret-lo"></span>
      <span class="cell-label">return address (low)</span>
    </div>
    <div class="row divider-row">
      <span class="addr">0x10</span>
      <span class="cell canary-cell" id="cell-canary"></span>
      <span class="cell-label">stack canary (0xDEAD)</span>
    </div>
    <div class="row" id="row-buf3">
      <span class="addr">0x0C</span>
      <span class="cell" id="cell-b3"></span>
      <span class="cell-label">buf [3]</span>
    </div>
    <div class="row" id="row-buf2">
      <span class="addr">0x08</span>
      <span class="cell" id="cell-b2"></span>
      <span class="cell-label">buf [2]</span>
    </div>
    <div class="row" id="row-buf1">
      <span class="addr">0x04</span>
      <span class="cell" id="cell-b1"></span>
      <span class="cell-label">buf [1]</span>
    </div>
    <div class="row" id="row-buf0">
      <span class="addr">0x00</span>
      <span class="cell" id="cell-b0"></span>
      <span class="cell-label">buf [0]</span>
    </div>
  </div>
  <div class="status-bar" id="status-bar"></div>
</div>

/* Base layout and input styles */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; padding: .5rem; }
.demo-wrap { max-width: 520px; margin: 0 auto; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .8rem; line-height: 1.5; }
.input-row { display: flex; gap: .5rem; margin-bottom: .9rem; }
input { flex: 1; padding: .4rem .7rem; font: 15px ui-monospace, monospace; border: 1px solid #ccc;
        border-radius: 6px; outline: none; }
input:focus { border-color: #1d3557; }
button { font: 600 13px system-ui; padding: .38rem .85rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 6px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
/* Stack diagram grid */
.stack-diagram { border: 1px solid #cdd9e3; border-radius: 8px; overflow: hidden; margin-bottom: .7rem; }
.row { display: grid; grid-template-columns: 56px 1fr 1fr; align-items: center;
       border-bottom: 1px solid #e4eaf0; }
.header-row { background: #f0f4f8; font-size: .78rem; font-weight: 700; text-transform: uppercase;
              color: #5a7088; letter-spacing: .04em; }
.header-row span { padding: .3rem .5rem; }
.row:last-child { border-bottom: none; }
.addr { font: 600 12px ui-monospace, monospace; color: #888; padding: .42rem .5rem; }
.label { padding: .3rem .5rem; }
.cell { font: 700 14px ui-monospace, monospace; padding: .42rem .5rem; min-height: 2.1rem;
        display: flex; align-items: center; transition: background .15s; border-right: 1px solid #e4eaf0; }
.cell-label { font-size: .78rem; color: #5a7088; padding: .42rem .5rem; }
/* Cell state colors */
.cell.safe { background: #e6f4ea; color: #0a7d33; }
.cell.overflow { background: #fff0b0; color: #7a5800; }
.cell.canary-cell { background: #eef0ff; color: #3a4aad; }
.cell.canary-smashed { background: #fde8e8; color: #c92f3c !important; }
.cell.ret-smashed { background: #c92f3c; color: #fff !important; }
.divider-row { background: #f5f6ff; }
.status-bar { font-size: .95rem; font-weight: 600; min-height: 1.5rem; margin-top: .2rem; }
.status-bar.ok { color: #0a7d33; }
.status-bar.warn { color: #7a5800; }
.status-bar.danger { color: #c92f3c; }

const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Secret sentinel placed by the compiler between buffer and return address
const CANARY = 0xDEAD;
// Stack layout (low address at top, high at bottom in this visual)
// buf[0..3], canary(word), ret-lo(word), ret-hi(word)
const SAFE_RET_LO = 0x4008;
const SAFE_RET_HI = 0x0000;

function toHex2(n) { return '0x' + (n & 0xFF).toString(16).toUpperCase().padStart(2, '0'); }
function toHex4(n) { return '0x' + (n & 0xFFFF).toString(16).toUpperCase().padStart(4, '0'); }

// Re-render every keystroke
function update() {
  const raw = document.getElementById('user-input').value;
  // Convert each character to its ASCII byte value
  const bytes = [];
  for (let i = 0; i < raw.length; i++) bytes.push(raw.charCodeAt(i) & 0xFF);

  // Fill the four buffer slots
  const bufCells = ['cell-b0', 'cell-b1', 'cell-b2', 'cell-b3'];
  for (let i = 0; i < 4; i++) {
    const el = document.getElementById(bufCells[i]);
    if (bytes[i] !== undefined) {
      el.textContent = toHex2(bytes[i]);
      el.className = 'cell safe';
    } else {
      el.textContent = '0x00';
      el.className = 'cell';
    }
  }

  // Canary slot — byte 4 (index 4) hits it
  const canaryEl = document.getElementById('cell-canary');
  const canaryByte = bytes[4];
  if (canaryByte !== undefined) {
    // Any byte that is not the expected high byte smashes the canary
    const smashed = (canaryByte !== ((CANARY >> 8) & 0xFF));
    canaryEl.textContent = smashed ? toHex2(canaryByte) : toHex4(CANARY);
    canaryEl.className = 'cell ' + (smashed ? 'canary-smashed' : 'canary-cell');
  } else {
    canaryEl.textContent = toHex4(CANARY);
    canaryEl.className = 'cell canary-cell';
  }

  // Bytes 5+ overwrite the return address
  const retLoEl = document.getElementById('cell-ret-lo');
  const retHiEl = document.getElementById('cell-ret-hi');
  const overflowByteLo = bytes[5];
  const overflowByteHi = bytes[6];
  if (overflowByteLo !== undefined) {
    const newLo = (overflowByteLo << 8) | (bytes[6] !== undefined ? bytes[6] : 0x08);
    retLoEl.textContent = toHex4(newLo);
    retLoEl.className = 'cell ret-smashed';
  } else {
    retLoEl.textContent = toHex4(SAFE_RET_LO);
    retLoEl.className = 'cell safe';
  }
  if (overflowByteHi !== undefined) {
    retHiEl.textContent = toHex2(overflowByteHi);
    retHiEl.className = 'cell ret-smashed';
  } else {
    retHiEl.textContent = toHex4(SAFE_RET_HI);
    retHiEl.className = 'cell safe';
  }

  // Status message
  const bar = document.getElementById('status-bar');
  const len = bytes.length;
  if (len === 0) {
    bar.textContent = 'Start typing to fill the buffer.';
    bar.className = 'status-bar';
  } else if (len <= 4) {
    bar.textContent = i18nFmt('%n% byte(s) written — buffer intact.', { n: len });
    bar.className = 'status-bar ok';
  } else if (len === 5) {
    bar.textContent = 'Byte 5 hit the canary! A real program would abort here.';
    bar.className = 'status-bar warn';
  } else {
    bar.textContent = 'Return address corrupted — execution would be hijacked!';
    bar.className = 'status-bar danger';
  }
}

document.getElementById('user-input').addEventListener('input', update);
document.getElementById('btn-reset').addEventListener('click', () => {
  document.getElementById('user-input').value = '';
  update();
});
update();
Result
Console

Notice what happens as you cross the buffer boundary. The first overflow byte hits the canary (a secret sentinel value placed by the compiler between the buffer and the return address). A real program checks the canary before returning; if it has changed, the process is killed immediately. Keep writing and you reach the return address itself — which is what a real exploit would overwrite to redirect execution.

The Real Complexity

A raw overflow is easy to stop — or so it seemed. Three major defenses were added, each defeated by a new attack class:

  • Stack canaries (GCC -fstack-protector, 1998): the compiler inserts a random secret word between the buffer and the return address. Before returning, the runtime checks that the canary is intact. Bypassed by format-string leaks that reveal the canary value before the overflow.
  • Non-executable stack / NX bit / DEP (hardware, early 2000s): the CPU refuses to execute code in stack pages. Bypassed by return-to-libc: instead of injecting shellcode, the attacker overwrites the return address to point to an existing function such as system().
  • Address Space Layout Randomization (ASLR) (Linux 2.6.12, 2005): the OS places the stack, heap, and libraries at random addresses each run, so the attacker cannot hardcode targets. Partially bypassed by partial overwrites, heap sprays, and information-leak gadgets.
  • Return-Oriented Programming (ROP): chains tiny existing code snippets ("gadgets", each ending in a ret instruction) to build arbitrary computation without injecting any new code — bypassing NX and ASLR together when combined with a leak.

Today's production binaries use all three defenses plus Control-Flow Integrity (CFI), which restricts where indirect calls and returns may land. Yet fuzzing and formal verification continue to find new overflows, and the underlying problem — C and C++ do not enforce array bounds — remains.

The deeper issue relates to the halting problem: deciding statically whether every execution path of a program ever overflows a buffer is undecidable in general. Defenses are therefore probabilistic, not complete.

Where It Matters

Buffer overflows are not theoretical:

  • The Morris Worm (1988): the first self-propagating internet worm exploited a gets() overflow in fingerd. It infected roughly 6,000 machines — 10 % of the internet at the time.
  • CVE statistics: for decades, buffer overflows have ranked among the top three most common vulnerability classes in the US National Vulnerability Database.
  • Browser engines: Chrome, Firefox, and Safari have each shipped critical overflow patches. Modern browsers sandbox renderer processes partly to limit the damage of memory-corruption bugs.
  • Memory-safe languages: Rust, Go, Swift, and managed languages like Java enforce bounds checks at compile or run time, eliminating this class of bug entirely — at the cost of some performance. Microsoft has stated that 70 % of its security patches address memory-safety issues.
  • Formal verification: tools like SAT-based model checkers and abstract interpreters can prove the absence of overflows in small, critical modules.

The lesson is stark: languages and runtimes that enforce memory safety eliminate buffer overflows by design. The ongoing migration of systems software from C/C++ to Rust is largely motivated by this single fact.

Conclusion

A buffer overflow reduces to the simplest possible mistake: writing one byte too many. Yet that one byte can reach the saved return address and hand control of the entire program to an attacker. Stack canaries, ASLR, and non-executable stacks each raise the cost of exploitation without eliminating the underlying cause.

The only complete fix is to use a language that never lets the mistake happen in the first place. Until every critical system is written in memory-safe code, the stack will keep reminding us that bounds checking is not optional — and that the halting problem means no static tool can catch every instance automatically.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/buffer-overflow/Content licensed under CC BY-NC 4.0.