The ZAB Protocol

Distributed systems live and die by agreement. When thousands of microservices share configuration, locks, or leader elections through Apache ZooKeeper, every one of them must see the exact same history in the exact same order — even if the machine running the show just crashed and rebooted.

That guarantee is delivered by ZAB (ZooKeeper Atomic Broadcast), a protocol designed by Flavio Junqueira, Benjamin Reed, and Marco Serafini and published in 2011. ZAB is not a general-purpose consensus algorithm: it is tuned for a primary-backup model where one leader serializes all writes and followers replicate them in order.

The challenge is recovery. A new leader inherits an ensemble of followers that may each have seen a different prefix of history. ZAB's recovery phase stitches those prefixes together so the new leader — and every follower — starts from a consistent, agreed-upon state before accepting a single new write. The entire system then resumes as if nothing happened.

The cluster below has one leader (L) and two followers (F1, F2). Broadcast messages one at a time to see how the leader sends a PROPOSAL, waits for a quorum ACK, then sends COMMIT — ensuring every follower delivers messages in the same order.

<!-- {{c_layout_comment}} -->
<div class="hint-box">{{hint_intro}}</div>
<div id="cluster" class="cluster">
  <div class="node leader" id="node-L">
    <span class="node-label">{{label_leader}}</span>
    <span class="node-zxid" id="zxid-L">zxid: 0</span>
  </div>
  <div class="node follower" id="node-F1">
    <span class="node-label">F1</span>
    <span class="node-zxid" id="zxid-F1">zxid: 0</span>
  </div>
  <div class="node follower" id="node-F2">
    <span class="node-label">F2</span>
    <span class="node-zxid" id="zxid-F2">zxid: 0</span>
  </div>
</div>
<div id="log-area" class="log-area"></div>
<div class="btns">
  <button id="btn-broadcast" type="button">{{btn_broadcast}}</button>
  <button id="btn-crash" type="button" class="danger">{{btn_crash}}</button>
  <button id="btn-reset" type="button" class="ghost">{{btn_reset}}</button>
</div>

/* {{c_css_comment}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint-box { font-size: .87rem; color: #444; margin-bottom: .7rem; line-height: 1.45; }
.cluster { display: flex; gap: 12px; margin-bottom: .6rem; flex-wrap: wrap; }
.node { display: flex; flex-direction: column; align-items: center; justify-content: center;
        width: 80px; height: 80px; border-radius: 12px; border: 2px solid #adb1b8;
        background: #e8eef3; transition: background .25s, border-color .25s; }
.node.leader { border-color: #1d3557; background: #1d3557; color: #fff; }
.node.crashed { border-color: #c92f3c; background: #fde8ea; color: #c92f3c; opacity: .7; }
.node.recovering { border-color: #e67e22; background: #fef3e2; color: #c0661e; }
.node-label { font-weight: 700; font-size: 1.05rem; }
.node-zxid { font-size: .72rem; margin-top: 3px; opacity: .85; }
.log-area { border: 1px solid #cdd9e3; border-radius: 8px; background: #f7f9fb;
            min-height: 120px; max-height: 160px; overflow-y: auto;
            padding: .5rem .7rem; font-size: .82rem; margin-bottom: .6rem; }
.log-entry { margin: 2px 0; line-height: 1.4; }
.log-entry.phase-propose { color: #1d3557; }
.log-entry.phase-ack    { color: #0a7d33; }
.log-entry.phase-commit { color: #0a7d33; font-weight: 600; }
.log-entry.phase-crash  { color: #c92f3c; font-weight: 700; }
.log-entry.phase-recover { color: #c0661e; }
.log-entry.phase-info   { color: #555; font-style: italic; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.danger { background: #c92f3c; border-color: #c92f3c; }
button.ghost  { background: #fff; color: #1d3557; }

// Code not found

When you crash the leader, a new leader is elected and runs ZAB's recovery phase: it synchronizes the highest committed transaction ID (zxid) across the surviving followers before any new message can be broadcast. Notice that messages in-flight during the crash are handled safely — ZAB's primary-order guarantee means a committed message is never lost and a not-yet-committed message is never half-delivered.

Atomic broadcast says: all correct processes deliver the same messages in the same order. ZAB strengthens this with primary-order, a property invented for the primary-backup model:

• Total order: if server $A$ delivers transaction $T_1$ before $T_2$, then every server that delivers both delivers them in the same order.
• Primary order: if a primary $p_1$ broadcasts $T_1$ before handing leadership to $p_2$, and $p_2$ later broadcasts $T_2$, then every server delivers $T_1$ before $T_2$.
• Primary integrity: $p_2$ delivers all transactions broadcast by $p_1$ before broadcasting its first own transaction.

These guarantees ride on two mechanisms. Every transaction carries a zxid — a 64-bit counter split into an epoch (upper 32 bits, incremented at each leader change) and a counter (lower 32 bits, reset to zero each epoch). A lexicographic comparison of zxids gives a total order across epochs. The two-phase broadcast (PROPOSAL → quorum ACK → COMMIT) ensures a message is only delivered after a quorum has acknowledged it, so no single failure can erase a committed write.

Recovery adds a third phase. The new leader collects the zxid histories of a quorum of followers, identifies the highest committed zxid among them, replicates any missing transactions to every follower, and only then broadcasts an epoch-bump NEW_EPOCH message. From that point the system re-enters normal operation. The whole recovery is bounded by the size of the backlog — $O(n)$ messages where $n$ is the number of un-delivered transactions — not by the total history.

Compare this with Raft: Raft is a general-purpose consensus algorithm that also handles leader election, log compaction and membership changes; ZAB is purpose-built for the primary-backup model and trades generality for a tighter primary-order contract. Both sit squarely in the family of problems related to distributed agreement — solvable in practice, but never in a system where nodes can fail arbitrarily without any timing assumptions at all.

ZooKeeper's promise — "give me a consistent, ordered log of configuration changes, and I'll give you a solid foundation for anything distributed" — is only as strong as ZAB. The protocol shows up wherever systems need a single source of truth that survives crashes:

• Hadoop NameNode HA: two NameNodes use ZooKeeper (via ZAB) to elect the active node and prevent split-brain; the standby can take over without losing any committed metadata.
• Apache Kafka (pre-KRaft): Kafka relied on ZooKeeper for broker and controller elections; every topic partition leader change was mediated by ZAB-ordered writes.
• HBase: the HBase master uses ZooKeeper sessions as failure detectors; a crashed master is detected in milliseconds because its ZooKeeper session expires, and a new master is elected without any manual intervention.
• Apache Mesos / Kubernetes etcd: etcd uses its own Raft-based protocol, but the design choices mirror ZAB's; any cluster that needs "one leader, everyone else replicates" owes a debt to the primary-order model ZAB formalized.

The deeper lesson is that ordering is the hard part of distribution. Once you have a reliable total-order broadcast, building locks, counters, barriers, and leader elections becomes almost mechanical. ZAB turned that insight into production-grade software.

ZAB is not the flashiest distributed-systems protocol, but it is one of the most carefully engineered. Its primary-order guarantee — every committed message from the old leader reaches every server before the new leader speaks — is exactly the contract that makes ZooKeeper safe to use as a coordination primitive for much larger systems.

The next time a Kafka broker election completes in under a second, or a Hadoop NameNode failover preserves every committed block mapping, you can thank ZAB's two-phase broadcast and its recovery-phase zxid synchronization for keeping the order intact. Distributed agreement is hard in theory; ZAB is proof that it can be made reliable in practice.