Buffer Overflow

Every function call in a C-style program uses the stack: a strip of memory that grows downward. When foo() calls bar(), the CPU pushes a return address — the instruction to jump back to when bar finishes — right next to the local variables of bar.

Most of those local variables are buffers: fixed-size byte arrays meant to hold input. The classic C functions gets, strcpy, and scanf copy bytes into a buffer without checking its size. Write more bytes than the buffer holds and they spill into adjacent memory — first into other local variables, and then, crucially, into the saved return address.

Overwrite that address with a value the attacker controls, and the CPU will "return" to whatever instruction they choose. That is a buffer overflow exploit: a bug so reliable and so severe that it shaped the entire history of systems security.

The mechanism was known by the 1970s, but the landmark paper by Aleph One, Smashing the Stack for Fun and Profit (Phrack, 1996), made it accessible to a generation of security researchers and attackers alike. The story since then is a race between increasingly clever exploits and increasingly robust defenses.

The demo below shows a simplified stack frame with a 4-byte buffer, a stack canary, and a saved return address. Type characters in the input field and watch the bytes fill the buffer — and then spill beyond it.

<!-- {{c_layout_comment}} -->
<div class="demo-wrap">
  <p class="hint">{{hint_para}}</p>
  <div class="input-row">
    <input id="user-input" type="text" maxlength="16" placeholder="{{input_placeholder}}" autocomplete="off" spellcheck="false">
    <button id="btn-reset" type="button" class="ghost">{{btn_reset}}</button>
  </div>
  <!-- {{c_stack_comment}} -->
  <div class="stack-diagram" id="stack-diagram" aria-label="{{stack_aria}}">
    <div class="row header-row">
      <span class="label">{{label_address}}</span>
      <span class="label">{{label_cell}}</span>
      <span class="label">{{label_status}}</span>
    </div>
    <div class="row" id="row-ret-hi">
      <span class="addr">0x18</span>
      <span class="cell" id="cell-ret-hi"></span>
      <span class="cell-label">{{cell_ret_hi}}</span>
    </div>
    <div class="row" id="row-ret-lo">
      <span class="addr">0x14</span>
      <span class="cell" id="cell-ret-lo"></span>
      <span class="cell-label">{{cell_ret_lo}}</span>
    </div>
    <div class="row divider-row">
      <span class="addr">0x10</span>
      <span class="cell canary-cell" id="cell-canary"></span>
      <span class="cell-label">{{cell_canary_label}}</span>
    </div>
    <div class="row" id="row-buf3">
      <span class="addr">0x0C</span>
      <span class="cell" id="cell-b3"></span>
      <span class="cell-label">{{cell_buf}} [3]</span>
    </div>
    <div class="row" id="row-buf2">
      <span class="addr">0x08</span>
      <span class="cell" id="cell-b2"></span>
      <span class="cell-label">{{cell_buf}} [2]</span>
    </div>
    <div class="row" id="row-buf1">
      <span class="addr">0x04</span>
      <span class="cell" id="cell-b1"></span>
      <span class="cell-label">{{cell_buf}} [1]</span>
    </div>
    <div class="row" id="row-buf0">
      <span class="addr">0x00</span>
      <span class="cell" id="cell-b0"></span>
      <span class="cell-label">{{cell_buf}} [0]</span>
    </div>
  </div>
  <div class="status-bar" id="status-bar"></div>
</div>

/* {{c_base_styles}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; padding: .5rem; }
.demo-wrap { max-width: 520px; margin: 0 auto; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .8rem; line-height: 1.5; }
.input-row { display: flex; gap: .5rem; margin-bottom: .9rem; }
input { flex: 1; padding: .4rem .7rem; font: 15px ui-monospace, monospace; border: 1px solid #ccc;
        border-radius: 6px; outline: none; }
input:focus { border-color: #1d3557; }
button { font: 600 13px system-ui; padding: .38rem .85rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 6px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
/* {{c_stack_styles}} */
.stack-diagram { border: 1px solid #cdd9e3; border-radius: 8px; overflow: hidden; margin-bottom: .7rem; }
.row { display: grid; grid-template-columns: 56px 1fr 1fr; align-items: center;
       border-bottom: 1px solid #e4eaf0; }
.header-row { background: #f0f4f8; font-size: .78rem; font-weight: 700; text-transform: uppercase;
              color: #5a7088; letter-spacing: .04em; }
.header-row span { padding: .3rem .5rem; }
.row:last-child { border-bottom: none; }
.addr { font: 600 12px ui-monospace, monospace; color: #888; padding: .42rem .5rem; }
.label { padding: .3rem .5rem; }
.cell { font: 700 14px ui-monospace, monospace; padding: .42rem .5rem; min-height: 2.1rem;
        display: flex; align-items: center; transition: background .15s; border-right: 1px solid #e4eaf0; }
.cell-label { font-size: .78rem; color: #5a7088; padding: .42rem .5rem; }
/* {{c_state_styles}} */
.cell.safe { background: #e6f4ea; color: #0a7d33; }
.cell.overflow { background: #fff0b0; color: #7a5800; }
.cell.canary-cell { background: #eef0ff; color: #3a4aad; }
.cell.canary-smashed { background: #fde8e8; color: #c92f3c !important; }
.cell.ret-smashed { background: #c92f3c; color: #fff !important; }
.divider-row { background: #f5f6ff; }
.status-bar { font-size: .95rem; font-weight: 600; min-height: 1.5rem; margin-top: .2rem; }
.status-bar.ok { color: #0a7d33; }
.status-bar.warn { color: #7a5800; }
.status-bar.danger { color: #c92f3c; }

// Code not found

Notice what happens as you cross the buffer boundary. The first overflow byte hits the canary (a secret sentinel value placed by the compiler between the buffer and the return address). A real program checks the canary before returning; if it has changed, the process is killed immediately. Keep writing and you reach the return address itself — which is what a real exploit would overwrite to redirect execution.

A raw overflow is easy to stop — or so it seemed. Three major defenses were added, each defeated by a new attack class:

• Stack canaries (GCC -fstack-protector, 1998): the compiler inserts a random secret word between the buffer and the return address. Before returning, the runtime checks that the canary is intact. Bypassed by format-string leaks that reveal the canary value before the overflow.
• Non-executable stack / NX bit / DEP (hardware, early 2000s): the CPU refuses to execute code in stack pages. Bypassed by return-to-libc: instead of injecting shellcode, the attacker overwrites the return address to point to an existing function such as system().
• Address Space Layout Randomization (ASLR) (Linux 2.6.12, 2005): the OS places the stack, heap, and libraries at random addresses each run, so the attacker cannot hardcode targets. Partially bypassed by partial overwrites, heap sprays, and information-leak gadgets.
• Return-Oriented Programming (ROP): chains tiny existing code snippets ("gadgets", each ending in a ret instruction) to build arbitrary computation without injecting any new code — bypassing NX and ASLR together when combined with a leak.

Today's production binaries use all three defenses plus Control-Flow Integrity (CFI), which restricts where indirect calls and returns may land. Yet fuzzing and formal verification continue to find new overflows, and the underlying problem — C and C++ do not enforce array bounds — remains.

The deeper issue relates to the halting problem: deciding statically whether every execution path of a program ever overflows a buffer is undecidable in general. Defenses are therefore probabilistic, not complete.

Buffer overflows are not theoretical:

• The Morris Worm (1988): the first self-propagating internet worm exploited a gets() overflow in fingerd. It infected roughly 6,000 machines — 10 % of the internet at the time.
• CVE statistics: for decades, buffer overflows have ranked among the top three most common vulnerability classes in the US National Vulnerability Database.
• Browser engines: Chrome, Firefox, and Safari have each shipped critical overflow patches. Modern browsers sandbox renderer processes partly to limit the damage of memory-corruption bugs.
• Memory-safe languages: Rust, Go, Swift, and managed languages like Java enforce bounds checks at compile or run time, eliminating this class of bug entirely — at the cost of some performance. Microsoft has stated that 70 % of its security patches address memory-safety issues.
• Formal verification: tools like SAT-based model checkers and abstract interpreters can prove the absence of overflows in small, critical modules.

The lesson is stark: languages and runtimes that enforce memory safety eliminate buffer overflows by design. The ongoing migration of systems software from C/C++ to Rust is largely motivated by this single fact.

A buffer overflow reduces to the simplest possible mistake: writing one byte too many. Yet that one byte can reach the saved return address and hand control of the entire program to an attacker. Stack canaries, ASLR, and non-executable stacks each raise the cost of exploitation without eliminating the underlying cause.

The only complete fix is to use a language that never lets the mistake happen in the first place. Until every critical system is written in memory-safe code, the stack will keep reminding us that bounds checking is not optional — and that the halting problem means no static tool can catch every instance automatically.