Zookeeper 3.4.x Election Port 3888 Failure: Root Cause and Fix

Conclusion

Zookeeper versions ≤3.4.6 have a critical bug where the default election port 3888 can become non‑functional, preventing leader election. The related issues are Zookeeper‑3016 and Zookeeper‑2186.

Problem Background

On 20 December, two nodes in a five‑node Zookeeper cluster could not join the ensemble after a normal restart, while the remaining nodes continued read/write operations.

Symptoms

Environment

zoo.cfg contains five servers with ports 2888:3888.

Observed behavior

Servers 6 and 8 restarted but failed to join.

Servers 7, 9, and 10 (leader) remained operational.

Investigation

10’s port 3888 status

netstat shows LISTEN with many CLOSE_WAIT connections; telnet cannot establish a connection, only SYN_SENT appears.

jstack on server 10

Comparison with another team’s stack revealed the missing QuorumCnxManager$Listener thread responsible for accepting election requests.

Server 6 port 3888

Similar telnet failure and logs show a NegativeArraySizeException, causing the listener thread to die.

Root Cause Analysis

Online verification

The exception originates from QuorumCnxManager$Listener.receiveConnection() reading a negative num_remaining_bytes from a malformed packet, which triggers NegativeArraySizeException. The malformed packets were generated by a security‑scanning tool (IP 10.177).

Version fix

Issue‑2186 indicates the bug is fixed in Zookeeper 3.4.7; upgrading from 3.4.6 resolves the problem.

Summary

Unexpected issues can reveal hidden weaknesses; thorough investigation is essential.

Consulting knowledgeable peers accelerates problem resolution.

Upgrading to a newer Zookeeper version eliminates the election‑port failure.

References

issue‑3016: https://issues.apache.org/jira/browse/ZOOKEEPER-3016

issue‑2186: https://issues.apache.org/jira/browse/ZOOKEEPER-2186