Introduction

Distributed systems live on many machines with many clocks, and those clocks are never perfectly in sync. That gap creates two hard problems that pull in opposite directions.

Lamport clocks (1978) guarantee causal consistency: if event AA caused event BB, then ts(A)<ts(B)\text{ts}(A) < \text{ts}(B). But a Lamport timestamp is just a counter — it drifts arbitrarily far from wall-clock time, so you cannot use it to answer questions like "give me everything that happened in the last five minutes."

Physical clocks (NTP, GPS, TrueTime) stay close to real time. But clock skew means two independent nodes can easily assign the same or even reversed timestamps to causally related events.

Hybrid Logical Clocks (HLC), introduced by Kulkarni, Demirbas, Madeppa, Avva and Bharadwaj in 2014, pack both worlds into a single compact timestamp: a physical component ll that tracks the maximum physical time seen so far, and a logical counter cc that breaks ties and advances only when the physical component cannot. The result is always within a bounded skew ε\varepsilon of true time — yet it respects causality just as faithfully as a Lamport clock.

The key invariant: for any two events ee and ff where e→fe \to f (e happened before f), hlc(e)<hlc(f)\text{hlc}(e) < \text{hlc}(f) — and l≤pt(now)+εl \leq \text{pt}(\text{now}) + \varepsilon where pt\text{pt} is physical time.

Try It

Three nodes exchange messages while their physical clocks drift independently. Watch how their HLC timestamps stay monotone and causally consistent even when the physical clocks slip backward or forward.

<!-- HLC demo layout: three node columns with physical clock and HLC display. -->
<div class="intro-text">Each node has a physical clock (pt) that may drift, and an HLC timestamp (l,c). Click local events or send messages between nodes to see how HLC stays monotone and causally consistent despite clock drift.</div>
<div id="network" class="network">
  <div class="node-col" id="col-A">
    <div class="node-label">Node A</div>
    <div class="clock-row"><span class="clock-badge">pt: <span id="pt-A">0</span></span><span class="clock-badge hlc">HLC: <span id="hlc-A">(0,0)</span></span></div>
    <div class="events" id="events-A"></div>
    <div class="node-btns">
      <button id="local-A" type="button">Local event</button>
      <button id="send-AB" type="button">Send → B</button>
      <button id="send-AC" type="button">Send → C</button>
    </div>
  </div>
  <div class="node-col" id="col-B">
    <div class="node-label">Node B</div>
    <div class="clock-row"><span class="clock-badge">pt: <span id="pt-B">0</span></span><span class="clock-badge hlc">HLC: <span id="hlc-B">(0,0)</span></span></div>
    <div class="events" id="events-B"></div>
    <div class="node-btns">
      <button id="local-B" type="button">Local event</button>
      <button id="send-BA" type="button">Send → A</button>
      <button id="send-BC" type="button">Send → C</button>
    </div>
  </div>
  <div class="node-col" id="col-C">
    <div class="node-label">Node C</div>
    <div class="clock-row"><span class="clock-badge">pt: <span id="pt-C">0</span></span><span class="clock-badge hlc">HLC: <span id="hlc-C">(0,0)</span></span></div>
    <div class="events" id="events-C"></div>
    <div class="node-btns">
      <button id="local-C" type="button">Local event</button>
      <button id="send-CA" type="button">Send → A</button>
      <button id="send-CB" type="button">Send → B</button>
    </div>
  </div>
</div>
<div class="status" id="status">Click events or send messages to observe HLC.</div>
<button id="drift-btn" type="button" class="drift-btn">Apply random drift</button>
<button id="reset-btn" type="button" class="ghost">Reset</button>

/* Layout and styling for the HLC three-node demo. */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; padding: 4px; }
.intro-text { font-size: .85rem; color: #444; margin-bottom: .6rem; line-height: 1.45; }
.network { display: flex; gap: 8px; align-items: flex-start; }
.node-col { flex: 1; background: #f0f4f8; border: 1px solid #cdd9e3; border-radius: 10px; padding: 8px; min-width: 0; }
.node-label { font-weight: 700; font-size: 1rem; text-align: center; margin-bottom: 4px; color: #1d3557; }
.clock-row { display: flex; gap: 4px; flex-wrap: wrap; margin-bottom: 6px; }
.clock-badge { font-size: .72rem; background: #dce8f3; border: 1px solid #b8ccdf; border-radius: 6px; padding: 2px 5px; color: #1d3557; }
.clock-badge.hlc { background: #d4edda; border-color: #9acba8; color: #0a5c2f; font-weight: 600; }
.events { min-height: 80px; max-height: 120px; overflow-y: auto; font-size: .75rem; margin-bottom: 6px; }
.event-entry { padding: 2px 0; border-bottom: 1px solid #e2e8ef; line-height: 1.35; }
.event-entry .etag { font-weight: 700; color: #1d3557; }
.event-entry .hlc-val { color: #0a5c2f; font-weight: 600; }
.event-entry .pt-val { color: #888; }
.event-entry.recv { background: #fff9e6; }
.node-btns { display: flex; flex-direction: column; gap: 4px; }
button { font: 600 12px system-ui, sans-serif; padding: .3rem .6rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; width: 100%; }
button.ghost { background: #fff; color: #1d3557; }
.drift-btn { background: #e63946; border-color: #c92f3c; margin-top: 6px; width: auto; padding: .35rem .9rem; }
.status { font-size: .85rem; font-weight: 600; margin: 6px 0 4px; min-height: 1.3em; color: #1d3557; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Hybrid Logical Clocks (HLC) demo — three nodes, physical clocks, causal ordering.
// Each node stores (l, c): l = max physical time seen; c = tie-breaking counter.

const nodes = ['A', 'B', 'C'];

// Initial state: physical time 10, logical time (0,0) for each node.
const state = {};
nodes.forEach(n => { state[n] = { pt: 10, l: 0, c: 0 }; });

// Random drift offsets applied to each node's physical clock.
// Positive = clock runs fast; negative = clock runs slow (or backward).
const drifts = { A: 0, B: 0, C: 0 };

function getPhysTime(node) {
  // Physical time = base counter + drift offset.
  return state[node].pt + drifts[node];
}

// Local / Send event: advance HLC according to the three-case rule.
function hlcLocal(node) {
  const pt = getPhysTime(node);
  const s = state[node];
  const lPrime = Math.max(s.l, pt);
  if (lPrime === s.l) {
    s.c += 1;
  } else {
    s.l = lPrime;
    s.c = 0;
  }
  s.pt += 1; // advance the base physical counter by 1 tick.
  return { l: s.l, c: s.c };
}

// Receive event: merge incoming (lm, cm) with local state.
function hlcReceive(node, lm, cm) {
  const pt = getPhysTime(node);
  const s = state[node];
  const lPrime = Math.max(s.l, lm, pt);
  if (lPrime === s.l && lPrime === lm) {
    s.c = Math.max(s.c, cm) + 1;
  } else if (lPrime === lm) {
    s.c = cm + 1;
  } else if (lPrime === s.l) {
    s.c = s.c + 1;
  } else {
    s.c = 0;
  }
  s.l = lPrime;
  s.pt += 1; // advance the base physical counter by 1 tick.
  return { l: s.l, c: s.c };
}

function fmtHLC(l, c) { return '(' + l + ',' + c + ')'; }

function addEvent(node, tag, hlc, ptVal, isRecv) {
  const div = document.createElement('div');
  div.className = 'event-entry' + (isRecv ? ' recv' : '');
  div.innerHTML = '<span class="etag">' + tag + '</span> '
    + '<span class="hlc-val">HLC=' + fmtHLC(hlc.l, hlc.c) + '</span> '
    + '<span class="pt-val">pt=' + ptVal + '</span>';
  const container = document.getElementById('events-' + node);
  container.appendChild(div);
  container.scrollTop = container.scrollHeight;
}

function updateDisplay(node) {
  const s = state[node];
  document.getElementById('pt-' + node).textContent = getPhysTime(node);
  document.getElementById('hlc-' + node).textContent = fmtHLC(s.l, s.c);
}

function doLocal(node) {
  const pt = getPhysTime(node);
  const hlc = hlcLocal(node);
  addEvent(node, 'local', hlc, pt, false);
  updateDisplay(node);
  setStatus(fmt('Node %node% local event → HLC=%fmtHLC%', { node: node, fmtHLC: fmtHLC(hlc.l, hlc.c) }));
}

function doSend(from, to) {
  const pt = getPhysTime(from);
  const hlc = hlcLocal(from);
  addEvent(from, fmt('send→%to%', { to: to }), hlc, pt, false);
  updateDisplay(from);
  // Message received — apply receive rule to destination node.
  const pt2 = getPhysTime(to);
  const hlc2 = hlcReceive(to, hlc.l, hlc.c);
  addEvent(to, fmt('recv←%from%', { from: from }), hlc2, pt2, true);
  updateDisplay(to);
  setStatus(fmt('Message %from%→%to%: HLC=%fmtHLC%', { from: from, to: to, fmtHLC: fmtHLC(hlc2.l, hlc2.c) }));
}

function setStatus(msg) {
  document.getElementById('status').textContent = msg;
}

// Apply random drift to each node's physical clock offset.
document.getElementById('drift-btn').addEventListener('click', () => {
  drifts.A += Math.floor(Math.random() * 5) - 2;
  drifts.B += Math.floor(Math.random() * 5) - 2;
  drifts.C += Math.floor(Math.random() * 5) - 2;
  nodes.forEach(updateDisplay);
  setStatus(fmt('Drift applied: A:%A% B:%B% C:%C%', { A: drifts.A, B: drifts.B, C: drifts.C }));
});

document.getElementById('reset-btn').addEventListener('click', () => {
  nodes.forEach(n => { state[n] = { pt: 10, l: 0, c: 0 }; drifts[n] = 0; });
  nodes.forEach(n => {
    document.getElementById('events-' + n).innerHTML = '';
    updateDisplay(n);
  });
  setStatus('Click events or send messages to observe HLC.');
});

// Wire all buttons for local events and sends.
nodes.forEach(n => {
  document.getElementById('local-' + n).addEventListener('click', () => doLocal(n));
});
[['A','B'],['A','C'],['B','A'],['B','C'],['C','A'],['C','B']].forEach(([f,t]) => {
  document.getElementById('send-' + f + t).addEventListener('click', () => doSend(f, t));
});

// Initialize display.
nodes.forEach(updateDisplay);
Result
Console

Notice that whenever a message arrives from a node with a higher physical component, the receiver's ll jumps up — and when two events share the same ll, the counter cc makes them strictly ordered. The physical clocks may disagree; the HLC timestamps never contradict causality.

The Real Complexity

The HLC timestamp is a pair (l,c)(l, c) where ll is the logical physical time (highest physical time seen) and cc is an integer counter that strictly orders events sharing the same ll.

The update rule has three cases:

  • Local event (no message): set l′=max⁡(l,pt)l' = \max(l, \text{pt}). If l′=ll' = l, increment cc; otherwise reset c=0c = 0.
  • Send event: same as local — produce (l′,c′)(l', c') and attach it to the message.
  • Receive event: given the incoming timestamp (lm,cm)(l_m, c_m), set l′=max⁡(l,lm,pt)l' = \max(l, l_m, \text{pt}). Then: if all three were equal, c′=max⁡(c,cm)+1c' = \max(c, c_m) + 1; if l′=lml' = l_m only, c′=cm+1c' = c_m + 1; if l′=ll' = l only, c′=c+1c' = c + 1; otherwise c′=0c' = 0.

Why it works:

  1. Causal consistency — every send/receive pair advances the timestamp, so the happened-before relation e→fe \to f implies hlc(e)<hlc(f)\text{hlc}(e) < \text{hlc}(f).
  2. Bounded physical drift — ll only increases, and it jumps to pt\text{pt} whenever pt\text{pt} overtakes it, so l≤pt+εl \leq \text{pt} + \varepsilon where ε\varepsilon is the maximum clock skew in the system.
  3. Compact representation — a 64-bit integer suffices: the top kk bits for ll (milliseconds since epoch) and the bottom 1616 bits for cc. The pair fits in the same space as a vanilla NTP timestamp.

The logical counter cc stays small in practice — it only grows if many events happen within the same millisecond on the same node, which is rare. Compare this to pure Lamport clocks: cc there can grow without bound, whereas in HLC it resets to 00 every time real time advances.

Where It Matters

HLC addresses a real pain point that arises any time you need to order events across data centers:

  • Distributed databases: CockroachDB, YugabyteDB and TiDB all use HLC (or a close variant) as the timestamp backbone for multi-version concurrency control. A consistent snapshot at time TT simply collects every version with l≤Tl \leq T — no waiting for a global oracle.
  • Causally consistent reads: a client that wrote at (l,c)(l, c) can send that timestamp to any replica and ask for the latest version at or after (l,c)(l, c). The replica waits until its HLC reaches that point — bounded by ε\varepsilon — then answers.
  • Event sourcing and audit logs: append-only logs that must be merged from multiple nodes can use HLC to produce a total order that is both human-readable (wall-clock proximity) and logically correct.
  • Conflict-free replicated data types (CRDTs): many CRDT implementations rely on timestamps to resolve concurrent updates; HLC gives them a timestamp that is simultaneously physical and causal.
  • Debugging distributed systems: because ll stays close to real time, log entries from different nodes can be compared by HLC and still line up sensibly with human-readable timestamps — unlike pure Lamport clocks.

Conclusion

The tension between physical time and logical causality is not a quirk of distributed systems — it is fundamental. Real clocks drift; causal order must be maintained. HLC resolves the tension with a two-field timestamp and a three-case update rule that fits in 64 bits.

The elegance is in what it avoids: no global time oracle, no expensive synchronization round-trips, no growing unbounded counters. Just a small increment to the physical timestamp when the world cooperates, and a logical tick when it does not.

The next time your distributed system needs to answer "what happened first?", remember that the answer sits in a pair (l,c)(l, c) — close enough to real time to be useful, logical enough to be correct.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/hybrid-logical-clocks/Content licensed under CC BY-NC 4.0.