How a Linux Kernel Bug Caused Random Java Microservice Disconnections

Background

The environment consisted of 11 Alibaba Cloud servers running a Spring Cloud Alibaba microservice cluster (Spring Boot 2.2.5, Spring Cloud Hoxton.SR3, Spring Cloud Alibaba 2.2.1, Java 8). Sixty microservices were registered to a single Nacos cluster. Traffic flowed through nginx → Spring Cloud Gateway → business services.

Incident

During a holiday, services randomly disappeared from Nacos (gateway reported "service not found"). The issue recurred every few days at random times; different services dropped each occurrence. Manual restart restored stability for 2–3 days.

Troubleshooting Process

Suspect 1: Server Memory Exhaustion

Alibaba Cloud console showed missing metrics (memory, CPU, load). free -m on the server showed normal memory. An Alibaba Cloud ticket was opened to fix the console; after repair, the disconnections persisted.

Suspect 2: CPU Saturation

top

showed normal CPU usage; command execution felt responsive.

Suspect 3: Disk Full

du -sh *

revealed ample disk space.

Suspect 4: Network Issues

Since services were dropped due to missing heartbeats, network was investigated. telnet, mtr -n, and netstat -nat | grep TIME_WAIT | wc -l gave only rough insights. Enabled tcp_tw_reuse via echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse. Nacos client logs showed no entries before dropout.

Suspect 5: Nacos Server Failure

Nacos server metrics (memory, CPU, disk) were normal; dozens of other services remained healthy. Nacos server logs confirmed active deregistration of the affected services — but only a subset on the same machine.

Suspect 6: Microservice Resource Overuse

Deployment scripts and load balancing were identical across servers; other instances of the same service ran fine. Increased heap size and added stack-trace printing; after waiting, the issue recurred with no stack traces (processes hadn't exited).

Deeper Investigation

Upgraded Nacos client (version 1.4.1 matched the Spring Cloud Alibaba version) — problem persisted. Used arthas to monitor metrics; all appeared normal. Hypothesized heartbeat thread was being killed. Examined Nacos client source (1.x heartbeat logic) and used Arthas to watch heartbeat calls — Arthas froze when the anomaly recurred, producing no output.

Built a custom monitor to catch dropouts during working hours. When the next event occurred:

Arthas could not attach to the JVM process. jstat showed normal GC. jmap / jstack failed to attach; used jstack -F -l <pid> > heap.txt (forced mode, less info). The 10,000+ line dump revealed no deadlocks or obvious anomalies.

During analysis, the monitor reported services had auto-recovered.

Repeated the capture on a second occurrence: same symptoms — Arthas attach failed, forced jstack succeeded, services self-recovered shortly after.

Root Cause Hypothesis & Confirmation

Concluded the JVM was experiencing a "fake death" (hang), freezing all threads including heartbeat and logging. Searched "JVM hang troubleshooting" and found a reference to a Linux kernel bug. Compared kernel versions across servers with uname -r; the faulty server ran an older kernel. Filed an Alibaba Cloud ticket to upgrade the kernel.

Resolution

After kernel upgrade and reboot, the issue disappeared over two days of observation. The root cause was a Linux kernel bug triggering JVM hangs, which silently stopped heartbeat threads and caused Nacos to deregister services.

Conclusion

The week-long debugging journey underscored the importance of systematic elimination, low-level tooling ( jstack -F, arthas), and considering OS/kernel layers when application-level diagnostics stall.