How Zookeeper Guarantees Reliable Session Management with Heartbeats

Anyone who has used Zookeeper knows that when creating a node you can choose between an ephemeral node and a persistent node . An ephemeral node is deleted when the client disconnects, while a persistent node remains regardless of the client’s connection state.

Ephemeral nodes are crucial for implementing distributed locks, leader election, and similar patterns, so Zookeeper must have an efficient session management mechanism that detects client session expiration, removes the client’s ephemeral nodes, and notifies other clients.

Zookeeper Heartbeat Mechanism

Why a Heartbeat Is Needed

Zookeeper supports two network libraries: a custom NIO implementation and Netty. Relying solely on TCP connection termination to detect client failures is unreliable because TCP cannot always promptly report a dead client, especially when the client crashes without sending a FIN packet or when the connection becomes idle for a long period.

Therefore, Zookeeper implements its own heartbeat protocol.

Client Sends Heartbeat

The client periodically sends a heartbeat packet to the server to indicate it is still alive, preventing the server from closing the connection. Any received packet, including regular create requests, is treated as a heartbeat.

Server Updates Session Information

When the server receives a packet, it calls SessionTracker.touchSession() to update the session. Each session is identified by a unique sessionId.

Challenges of Session Management

Requirements

Zookeeper needs a place to store sessions.

It must detect when a client has not sent a heartbeat for a configurable timeout.

Sessions are stored in a ConcurrentHashMap<Long, SessionImpl> called sessionsById.

Detecting Session Expiration

Zookeeper records the timestamp of the last received packet for each session. A periodic scan checks for sessions that have not received a packet within the configured timeout (e.g., 4 seconds).

Choosing a Scheduling Strategy

Using a simple fixed‑interval scheduler is insufficient because each session may have a different timeout and the next check time must be adjusted dynamically after each heartbeat.

Zookeeper Session Management Mechanism

The core mechanism consists of three steps:

1. Store expiration time and session set in expiryMap

The map’s key is the expiration timestamp (rounded to a fixed interval), and the value is a set of sessions that expire at that time.

2. Update expiryMap on each packet

When a packet arrives, Zookeeper calculates the new expiration bucket (using expirationInterval, which equals the configured tickTime, default 2 seconds) and moves the session to the appropriate bucket.

3. SessionTracker continuously scans expiryMap

A dedicated SessionTracker thread repeatedly obtains the next expiration key, sleeps until that time, and then expires all sessions in the corresponding bucket.

Code Example: Updating Expiry Queue

public Long update(E elem, int timeout) {
  Long prevExpiryTime = elemMap.get(elem);
  long now = Time.currentElapsedTime();
  Long newExpiryTime = roundToNextInterval(now + timeout);
  if (newExpiryTime.equals(prevExpiryTime)) {
    return null;
  }
  Set<E> set = expiryMap.get(newExpiryTime);
  if (set == null) {
    set = Collections.newSetFromMap(new ConcurrentHashMap<E, Boolean>());
    Set<E> existingSet = expiryMap.putIfAbsent(newExpiryTime, set);
    if (existingSet != null) {
      set = existingSet;
    }
  }
  set.add(elem);
  prevExpiryTime = elemMap.put(elem, newExpiryTime);
  if (prevExpiryTime != null && !newExpiryTime.equals(prevExpiryTime)) {
    Set<E> prevSet = expiryMap.get(prevExpiryTime);
    if (prevSet != null) {
      prevSet.remove(elem);
    }
  }
  return newExpiryTime;
}

Code Example: SessionTracker Loop

public void run() {
  try {
    while (running) {
      long waitTime = sessionExpiryQueue.getWaitTime();
      if (waitTime > 0) {
        Thread.sleep(waitTime);
        continue;
      }
      for (SessionImpl s : sessionExpiryQueue.poll()) {
        ServerMetrics.getMetrics().STALE_SESSIONS_EXPIRED.add(1);
        setSessionClosing(s.sessionId);
        expirer.expire(s);
      }
    }
  } catch (InterruptedException e) {
    handleException(this.getName(), e);
  }
  LOG.info("SessionTrackerImpl exited loop!");
}

In summary, Zookeeper stores each session in a map, updates its expiration bucket on every heartbeat, and a background thread efficiently expires sessions by scanning the bucketed expiryMap. This simple yet effective design enables reliable management of ephemeral nodes and other session‑dependent features.