Why Did My Nacos Client Spawn Hidden Threads? Debugging CPU Spikes in Java Services

Project Background

The problematic project connects to multiple Nacos instances across different namespaces, using HTTP client calls to Nacos APIs. The HTTP client itself is reliable, but the overall system experienced high CPU usage in the test environment.

Problem Diagnosis

First, the CPU usage was observed with top -Hp. The offending process ID was identified, and a thread dump was generated using jstack <pid> > 1.txt. The dump revealed many threads belonging to the Nacos client, such as:

"com.alibaba.nacos.client.config.security.updater" #2269 daemon prio=5 os_prio=0 tid=0x00007fa3ec401800 nid=0x8d85 waiting on condition
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   ...

These threads were internal to the Nacos client and were created unexpectedly.

Root Cause

Investigation of the source code showed that the thread pool is created in the ServerHttpAgent constructor:

// init executorService
this.executorService = new ScheduledThreadPoolExecutor(1, new ThreadFactory() {
    @Override
    public Thread newThread(Runnable r) {
        Thread t = new Thread(r);
        t.setName("com.alibaba.nacos.client.config.security.updater");
        t.setDaemon(true);
        return t;
    }
});

The constructor is invoked each time NacosFactory.createConfigService(properties) is called, causing a new thread to be started for every configuration service instance. This repeated creation leads to the observed CPU spike.

Raft Network Partition Handling

The article also discusses how Raft behaves when a node becomes network‑isolated. In a three‑node cluster, the isolated node repeatedly times out and increments its term, while the remaining nodes continue operating. After network recovery, the isolated node rejects RPCs with a smaller term, prompting the leader to increase its term and step down, followed by a new election.

To prevent safety violations during multiple voting rounds, Tendermint introduces a locking mechanism:

Prevote‑the‑Lock: Validators may only pre‑vote for blocks they are locked on, preventing conflicting pre‑commits.

Unlock‑on‑Polka: A lock is released only after seeing a higher‑round Polka, ensuring progress without compromising safety.

Another proposed solution replaces the single term with a tuple (term, nodeId), using lexicographic ordering to uniquely identify leaders and avoid Raft conflicts.

Proposed Fix

The fix implemented checks each Nacos config connection for liveness; if a connection is dead, it is shut down and a new one is created, avoiding the blanket recreation that spawns extra threads. After applying this change, CPU usage returned to normal in the test environment.