Bytecode Interpreters

When you run a Python script or a Java program, the machine never directly executes your source code. Instead, the language runtime compiles it to bytecode — a sequence of compact numeric opcodes — and then a loop called an interpreter steps through those opcodes one by one, executing each.

That loop sounds trivial. But it fires billions of times per second. Every dispatch — the act of reading an opcode and jumping to the right handler — is overhead the program cannot escape. Even a single extra branch-prediction miss per opcode can cut throughput in half.

The two classic strategies, switch dispatch and threaded dispatch, make very different promises to the CPU. Understanding why one is faster reveals something surprising: the bottleneck of an interpreter is not the work it does, but the overhead of deciding what to do next. That is a lesson that echoes all the way up to JIT compilers.

Below is a tiny virtual machine with four opcodes: PUSH, ADD, MUL, and HALT. Write a short program using those opcodes and run it under either dispatch strategy.

<!-- {{c_html_intro}} -->
<div class="toolbar">
  <label for="prog-select">{{lbl_program}}</label>
  <select id="prog-select">
    <option value="0">{{prog_0}}</option>
    <option value="1">{{prog_1}}</option>
    <option value="2">{{prog_2}}</option>
  </select>
  <label for="mode-select">{{lbl_mode}}</label>
  <select id="mode-select">
    <option value="switch">{{mode_switch}}</option>
    <option value="threaded">{{mode_threaded}}</option>
  </select>
  <button id="run-btn" type="button">{{btn_run}}</button>
  <button id="step-btn" type="button" class="ghost">{{btn_step}}</button>
  <button id="reset-btn" type="button" class="ghost">{{btn_reset}}</button>
</div>
<div class="panels">
  <div class="pane">
    <div class="pane-title">{{pane_program}}</div>
    <div id="prog-view" class="prog-view"></div>
  </div>
  <div class="pane">
    <div class="pane-title">{{pane_stack}}</div>
    <div id="stack-view" class="stack-view"><em>{{stack_empty}}</em></div>
  </div>
</div>
<div id="status" class="status">{{status_ready}}</div>
<div class="trace-label">{{lbl_trace}}</div>
<div id="trace" class="trace"></div>

/* {{c_css_layout}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; font-size: 14px; }
.toolbar { display: flex; flex-wrap: wrap; align-items: center; gap: .4rem .6rem; margin-bottom: .7rem; }
label { font-size: .8rem; color: #555; }
select { font-size: .85rem; padding: .25rem .4rem; border: 1px solid #bbb; border-radius: 6px; background: #fff; }
button { font: 600 13px system-ui, sans-serif; padding: .35rem .75rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
button:disabled { opacity: .45; cursor: default; }
.panels { display: flex; gap: .6rem; margin-bottom: .5rem; }
.pane { flex: 1; border: 1px solid #d0d7de; border-radius: 8px; overflow: hidden; min-width: 0; }
.pane-title { font-size: .75rem; font-weight: 700; text-transform: uppercase; letter-spacing: .04em;
              color: #555; background: #f0f4f8; padding: .3rem .6rem; border-bottom: 1px solid #d0d7de; }
/* {{c_css_prog}} */
.prog-view { padding: .4rem .5rem; font: 13px/1.7 ui-monospace, monospace; }
.instr { display: flex; align-items: center; gap: .4rem; padding: .1rem .3rem; border-radius: 5px; }
.instr.active { background: #fff3cd; }
.instr.done { opacity: .45; }
.ip-arrow { color: #e63946; font-size: 1rem; width: 1rem; flex-shrink: 0; }
.op-name { color: #1d3557; font-weight: 700; min-width: 3.5rem; }
.op-arg { color: #6a737d; }
/* {{c_css_stack}} */
.stack-view { padding: .5rem .6rem; font: 13px ui-monospace, monospace; min-height: 60px; }
.stack-item { background: #e8eef3; border: 1px solid #cdd9e3; border-radius: 5px;
              padding: .15rem .5rem; margin-bottom: .25rem; color: #1d3557; font-weight: 600; }
/* {{c_css_status}} */
.status { font-size: .95rem; font-weight: 600; margin: .35rem 0; min-height: 1.3em; color: #333; }
.status.ok { color: #0a7d33; }
.status.err { color: #c92f3c; }
.trace-label { font-size: .75rem; color: #555; font-weight: 700; text-transform: uppercase;
               letter-spacing: .04em; margin-top: .3rem; }
.trace { font: 11px/1.6 ui-monospace, monospace; color: #444; background: #f6f8fa;
         border: 1px solid #d0d7de; border-radius: 7px; padding: .4rem .6rem;
         max-height: 90px; overflow-y: auto; white-space: pre; }

// Code not found

Notice what changes between the two modes. Switch dispatch jumps back to the top of a single big switch after every opcode — one indirect branch per instruction that always targets the same place, making it easy to mispredict. Threaded dispatch stores the address of each handler directly in the instruction stream and jumps straight there — each branch target differs, letting the CPU predict each handler independently. On modern hardware that difference compounds across millions of instructions.

The performance gap between the two strategies comes from a single hardware fact: indirect branch prediction.

Modern CPUs predict the next instruction before the current one finishes. For a switch interpreter the "next instruction" is always the top of the switch — a single target, easy to predict, but wrong in a subtler sense: the CPU cannot know which case will be taken next. That inner branch still mispredicts frequently.

• Switch dispatch: one big loop, one switch, one indirect branch per opcode. The branch predictor sees the same site fire over and over with unpredictable targets. Mispredictions cost 10–20 cycles each on modern processors.
• Threaded dispatch: each handler ends with an indirect jump to the next handler's address, embedded in the instruction stream. Each jump site is unique, so the predictor builds a separate history per opcode pair. Ertl and Gregg (2003) measured roughly $2\times$ speedups over switch dispatch on real interpreters.
• Why not just JIT-compile? JIT compilation eliminates dispatch entirely — the native code for each opcode runs back-to-back without any jump overhead. But JIT adds warmup latency, memory, and engineering cost. Threaded dispatch is the sweet spot for interpreters that must start fast and run reasonably well.

CPython uses switch dispatch for portability (computed-goto threading is a GCC extension). The JVM's HotSpot uses a template interpreter — a form of threaded dispatch — and switches to JIT once a method runs enough times. The choice is never just about speed; it is about the full trade-off between portability, startup time, and peak throughput.

Bytecode interpreters are not an academic curiosity — they run the code of billions of devices:

• CPython: the reference Python interpreter uses a switch loop. The python -O flag and projects like PyPy experiment with alternative dispatch to squeeze out speed without full JIT complexity.
• JVM: HotSpot's template interpreter is a hand-written threaded interpreter in assembly. It handles every method call until the JIT kicks in, making its speed critical for startup-sensitive workloads.
• Ruby (YARV): Ruby's virtual machine uses direct threading with computed gotos (on supported compilers), giving it a meaningful edge over a naive switch loop.
• WebAssembly runtimes: engines like Wasmtime and V8's Liftoff tier use threaded or token-threaded approaches to get Wasm running fast before the optimizing compiler finishes.
• Embedded scripting: Lua, Wren, and similar engines embed in games and tools where a JIT is too heavy. Their tight threaded interpreters execute millions of simple game-logic instructions per frame.

The dispatch loop is also the entry point for security research: speculative execution attacks like Spectre exploit the very branch-prediction machinery that makes threaded dispatch fast. Understanding JIT compilation and automata minimization helps complete the picture of how language runtimes balance speed, safety, and correctness.

A bytecode interpreter looks like a trivial loop — read an opcode, do some work, repeat. But the dispatch step, the jump from one handler to the next, is where the CPU fights hardest. Switch dispatch sends every opcode through a single bottleneck; threaded dispatch spreads those jumps across independent branch sites that the predictor can learn separately.

The difference is not algorithmic complexity in the classical sense. It is the gap between what a program does and what the hardware has to predict in order to do it fast. That gap is why Python and Java feel slower than C even on identical algorithms, why the JVM bothers with a template interpreter before JIT, and why language runtime engineers study branch predictors as carefully as they study language semantics.

Next time your Python script feels slow, it is not always the algorithm. Sometimes it is billions of tiny mispredicted jumps, each costing twenty cycles — hiding in a loop you never see.