Introduction

Distributed systems live and die by agreement. When thousands of microservices share configuration, locks, or leader elections through Apache ZooKeeper, every one of them must see the exact same history in the exact same order — even if the machine running the show just crashed and rebooted.

That guarantee is delivered by ZAB (ZooKeeper Atomic Broadcast), a protocol designed by Flavio Junqueira, Benjamin Reed, and Marco Serafini and published in 2011. ZAB is not a general-purpose consensus algorithm: it is tuned for a primary-backup model where one leader serializes all writes and followers replicate them in order.

The challenge is recovery. A new leader inherits an ensemble of followers that may each have seen a different prefix of history. ZAB's recovery phase stitches those prefixes together so the new leader — and every follower — starts from a consistent, agreed-upon state before accepting a single new write. The entire system then resumes as if nothing happened.

Try It: Broadcast Under a Leader Change

The cluster below has one leader (L) and two followers (F1, F2). Broadcast messages one at a time to see how the leader sends a PROPOSAL, waits for a quorum ACK, then sends COMMIT — ensuring every follower delivers messages in the same order.

<!-- ZAB Protocol demo: cluster nodes + log + controls -->
<div class="hint-box">Broadcast messages to see ZAB’s two-phase protocol (PROPOSAL → ACK → COMMIT). Then crash the leader to watch the recovery phase synchronize zxids before the new leader accepts writes.</div>
<div id="cluster" class="cluster">
  <div class="node leader" id="node-L">
    <span class="node-label">L (leader)</span>
    <span class="node-zxid" id="zxid-L">zxid: 0</span>
  </div>
  <div class="node follower" id="node-F1">
    <span class="node-label">F1</span>
    <span class="node-zxid" id="zxid-F1">zxid: 0</span>
  </div>
  <div class="node follower" id="node-F2">
    <span class="node-label">F2</span>
    <span class="node-zxid" id="zxid-F2">zxid: 0</span>
  </div>
</div>
<div id="log-area" class="log-area"></div>
<div class="btns">
  <button id="btn-broadcast" type="button">Broadcast msg</button>
  <button id="btn-crash" type="button" class="danger">Crash leader</button>
  <button id="btn-reset" type="button" class="ghost">Reset</button>
</div>

/* ZAB demo styles */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint-box { font-size: .87rem; color: #444; margin-bottom: .7rem; line-height: 1.45; }
.cluster { display: flex; gap: 12px; margin-bottom: .6rem; flex-wrap: wrap; }
.node { display: flex; flex-direction: column; align-items: center; justify-content: center;
        width: 80px; height: 80px; border-radius: 12px; border: 2px solid #adb1b8;
        background: #e8eef3; transition: background .25s, border-color .25s; }
.node.leader { border-color: #1d3557; background: #1d3557; color: #fff; }
.node.crashed { border-color: #c92f3c; background: #fde8ea; color: #c92f3c; opacity: .7; }
.node.recovering { border-color: #e67e22; background: #fef3e2; color: #c0661e; }
.node-label { font-weight: 700; font-size: 1.05rem; }
.node-zxid { font-size: .72rem; margin-top: 3px; opacity: .85; }
.log-area { border: 1px solid #cdd9e3; border-radius: 8px; background: #f7f9fb;
            min-height: 120px; max-height: 160px; overflow-y: auto;
            padding: .5rem .7rem; font-size: .82rem; margin-bottom: .6rem; }
.log-entry { margin: 2px 0; line-height: 1.4; }
.log-entry.phase-propose { color: #1d3557; }
.log-entry.phase-ack    { color: #0a7d33; }
.log-entry.phase-commit { color: #0a7d33; font-weight: 600; }
.log-entry.phase-crash  { color: #c92f3c; font-weight: 700; }
.log-entry.phase-recover { color: #c0661e; }
.log-entry.phase-info   { color: #555; font-style: italic; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.danger { background: #c92f3c; border-color: #c92f3c; }
button.ghost  { background: #fff; color: #1d3557; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// ZAB (ZooKeeper Atomic Broadcast) demo — simplified two-phase broadcast + leader-change recovery.
// Phase 1: Leader sends PROPOSAL to all followers.
// Phase 2: Leader waits for quorum ACK, then sends COMMIT.

// Cluster state
let epoch = 1;
let counter = 0;
let leaderAlive = true;
let msgQueue = [];     // unused in this simplified model
let broadcasting = false;

// zxid encoding: epoch * 1000 + counter (simplified; real ZAB uses 32+32 bit split)
function makeZxid(ep, cnt) { return ep * 1000 + cnt; }
function zxidStr(ep, cnt) { return ep + '-' + cnt; }

// Node identifiers
const NODES = ['L', 'F1', 'F2'];
const zxids = { L: 0, F1: 0, F2: 0 };   // highest committed zxid per node

function updateNodeUI() {
  NODES.forEach(id => {
    const el = document.getElementById('node-' + id);
    const zxidEl = document.getElementById('zxid-' + id);
    if (!el) return;
    // update visual state
    if (id === 'L') {
      if (!leaderAlive) {
        el.className = 'node crashed';
        el.querySelector('.node-label').textContent = 'L ✗';
      } else {
        el.className = 'node leader';
        el.querySelector('.node-label').textContent = 'L (leader)';
      }
    }
    const ep = Math.floor(zxids[id] / 1000);
    const cnt = zxids[id] % 1000;
    zxidEl.textContent = 'zxid: ' + zxidStr(ep, cnt);
  });
}

function log(msg, phase) {
  const el = document.getElementById('log-area');
  const d = document.createElement('div');
  d.className = 'log-entry phase-' + phase;
  d.textContent = msg;
  el.appendChild(d);
  el.scrollTop = el.scrollHeight;
}

// Two-phase broadcast: PROPOSAL -> quorum ACK -> COMMIT
function broadcast() {
  if (!leaderAlive) { log('No leader — recovery in progress or cluster reset needed.', 'crash'); return; }
  if (broadcasting) return;
  broadcasting = true;
  counter++;
  const ep = epoch, cnt = counter;
  const zx = makeZxid(ep, cnt);
  const label = fmt('tx:%zxidStr%', { zxidStr: zxidStr(ep, cnt) });

  // Phase 1: PROPOSAL
  log('L → F1, F2: PROPOSAL(' + label + ')', 'phase-propose');

  // Phase 2: quorum ACK (both followers acknowledge)
  log('F1 → L: ACK(' + label + ')', 'phase-ack');
  log('F2 → L: ACK(' + label + ')', 'phase-ack');
  zxids['F1'] = zx;
  zxids['F2'] = zx;

  // Phase 3: COMMIT after quorum
  log('L → F1, F2: COMMIT(' + label + ')', 'phase-commit');
  zxids['L'] = zx;

  updateNodeUI();
  broadcasting = false;
}

// Crash the leader and run ZAB recovery
function crashLeader() {
  if (!leaderAlive) { log('Leader is already down.', 'crash'); return; }
  leaderAlive = false;
  log('⚡ Leader L crashed! Starting leader election and recovery...', 'crash');
  updateNodeUI();

  // Simulate recovery delay
  setTimeout(() => {
    log('Recovery phase: collecting highest zxid from quorum...', 'recover');

    // Find highest zxid in the surviving quorum
    const maxZx = Math.max(zxids['F1'], zxids['F2']);
    const maxEp = Math.floor(maxZx / 1000);
    const maxCnt = maxZx % 1000;
    log(fmt('Highest committed zxid in quorum: %zxidStr%', { zxidStr: zxidStr(maxEp, maxCnt) }), 'recover');

    // Bump epoch for new leader
    epoch++;
    counter = maxCnt;
    zxids['F1'] = makeZxid(epoch, counter);
    zxids['F2'] = makeZxid(epoch, counter);

    // F1 becomes new leader (simplified: reuse L node visually)
    const leaderEl = document.getElementById('node-L');
    leaderEl.className = 'node recovering';
    leaderEl.querySelector('.node-label').textContent = 'F1*';
    const zxEl = document.getElementById('zxid-L');
    zxids['L'] = makeZxid(epoch, counter);
    zxEl.textContent = 'zxid: ' + zxidStr(epoch, counter);

    log(fmt('NEW_EPOCH(%epoch%) → all followers synchronized', { epoch: epoch }), 'recover');
    log('Recovery complete. New leader is ready to accept writes.', 'phase-info');
    leaderAlive = true;
    NODES.forEach(id => {
      const el = document.getElementById('node-' + id);
      if (!el) return;
      if (el.classList.contains('crashed')) el.className = 'node follower';
      if (el.classList.contains('recovering')) el.className = 'node leader';
    });
    updateNodeUI();
  }, 800);
}

function reset() {
  epoch = 1; counter = 0; leaderAlive = true; broadcasting = false;
  NODES.forEach(id => { zxids[id] = 0; });
  document.getElementById('log-area').innerHTML = '';
  const leaderEl = document.getElementById('node-L');
  leaderEl.className = 'node leader';
  leaderEl.querySelector('.node-label').textContent = 'L (leader)';
  updateNodeUI();
  log('Cluster reset. Epoch 1, all zxids at 0.', 'phase-info');
}

document.getElementById('btn-broadcast').addEventListener('click', broadcast);
document.getElementById('btn-crash').addEventListener('click', crashLeader);
document.getElementById('btn-reset').addEventListener('click', reset);
updateNodeUI();
log('Cluster ready. Click ’Broadcast msg’ to start.', 'phase-info');
Result
Console

When you crash the leader, a new leader is elected and runs ZAB's recovery phase: it synchronizes the highest committed transaction ID (zxid) across the surviving followers before any new message can be broadcast. Notice that messages in-flight during the crash are handled safely — ZAB's primary-order guarantee means a committed message is never lost and a not-yet-committed message is never half-delivered.

The Real Complexity

Atomic broadcast says: all correct processes deliver the same messages in the same order. ZAB strengthens this with primary-order, a property invented for the primary-backup model:

  • Total order: if server AA delivers transaction T1T_1 before T2T_2, then every server that delivers both delivers them in the same order.
  • Primary order: if a primary p1p_1 broadcasts T1T_1 before handing leadership to p2p_2, and p2p_2 later broadcasts T2T_2, then every server delivers T1T_1 before T2T_2.
  • Primary integrity: p2p_2 delivers all transactions broadcast by p1p_1 before broadcasting its first own transaction.

These guarantees ride on two mechanisms. Every transaction carries a zxid — a 64-bit counter split into an epoch (upper 32 bits, incremented at each leader change) and a counter (lower 32 bits, reset to zero each epoch). A lexicographic comparison of zxids gives a total order across epochs. The two-phase broadcast (PROPOSAL → quorum ACK → COMMIT) ensures a message is only delivered after a quorum has acknowledged it, so no single failure can erase a committed write.

Recovery adds a third phase. The new leader collects the zxid histories of a quorum of followers, identifies the highest committed zxid among them, replicates any missing transactions to every follower, and only then broadcasts an epoch-bump NEW_EPOCH message. From that point the system re-enters normal operation. The whole recovery is bounded by the size of the backlog — O(n)O(n) messages where nn is the number of un-delivered transactions — not by the total history.

Compare this with Raft: Raft is a general-purpose consensus algorithm that also handles leader election, log compaction and membership changes; ZAB is purpose-built for the primary-backup model and trades generality for a tighter primary-order contract. Both sit squarely in the family of problems related to distributed agreement — solvable in practice, but never in a system where nodes can fail arbitrarily without any timing assumptions at all.

Where It Matters

ZooKeeper's promise — "give me a consistent, ordered log of configuration changes, and I'll give you a solid foundation for anything distributed" — is only as strong as ZAB. The protocol shows up wherever systems need a single source of truth that survives crashes:

  • Hadoop NameNode HA: two NameNodes use ZooKeeper (via ZAB) to elect the active node and prevent split-brain; the standby can take over without losing any committed metadata.
  • Apache Kafka (pre-KRaft): Kafka relied on ZooKeeper for broker and controller elections; every topic partition leader change was mediated by ZAB-ordered writes.
  • HBase: the HBase master uses ZooKeeper sessions as failure detectors; a crashed master is detected in milliseconds because its ZooKeeper session expires, and a new master is elected without any manual intervention.
  • Apache Mesos / Kubernetes etcd: etcd uses its own Raft-based protocol, but the design choices mirror ZAB's; any cluster that needs "one leader, everyone else replicates" owes a debt to the primary-order model ZAB formalized.

The deeper lesson is that ordering is the hard part of distribution. Once you have a reliable total-order broadcast, building locks, counters, barriers, and leader elections becomes almost mechanical. ZAB turned that insight into production-grade software.

Conclusion

ZAB is not the flashiest distributed-systems protocol, but it is one of the most carefully engineered. Its primary-order guarantee — every committed message from the old leader reaches every server before the new leader speaks — is exactly the contract that makes ZooKeeper safe to use as a coordination primitive for much larger systems.

The next time a Kafka broker election completes in under a second, or a Hadoop NameNode failover preserves every committed block mapping, you can thank ZAB's two-phase broadcast and its recovery-phase zxid synchronization for keeping the order intact. Distributed agreement is hard in theory; ZAB is proof that it can be made reliable in practice.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/zab-protocol/Content licensed under CC BY-NC 4.0.