Skip to content
Why Java Microservices Randomly Disappeared: A Linux Kernel Bug Hunt

Why Java Microservices Randomly Disappeared: A Linux Kernel Bug Hunt

The author details a week-long investigation into random microservice disconnections in a Spring Cloud Alibaba cluster, systematically ruling out memory, CPU, disk, network, Nacos server, and client versions before discovering a Linux kernel bug causing JVM hangs that silently dropped heartbeats, resolved by upgrading the kernel.

IoT Full-Stack Technology
IoT Full-Stack Technology
IoT Full-Stack Technology

Background

The environment consists of 11 Alibaba Cloud servers running a Spring Cloud Alibaba microservice cluster with 60 services registered to a single Nacos cluster. Traffic flows through nginx → Spring Cloud Gateway → business microservices. Key versions: Spring Boot 2.2.5.RELEASE, Spring Cloud Hoxton.SR3, Spring Cloud Alibaba 2.2.1.RELEASE (Nacos client 1.4.1), Java 1.8.

Incident

During a holiday, the gateway reported "service not found" errors. The Nacos dashboard showed random services disappearing (marked offline). The issue recurred every few days at random times, affecting different services each time. Manual kill+restart of affected services restored stability for 2–3 days.

Troubleshooting Process

Suspect 1: Server Memory Exhaustion

Alibaba Cloud console showed missing metrics (memory, CPU, load) for the faulty machine. free -m on the server showed normal memory. An Alibaba Cloud ticket was filed to fix the console display; after the fix, the disconnection issue persisted.

Suspect 2: CPU Saturation

Command execution felt responsive; top showed normal CPU usage.

Suspect 3: Disk Full

du -sh *

revealed ample disk space.

Suspect 4: Network Issues

Since service dropout implies missed heartbeats, network became the prime suspect. Tools used: telnet, mtr -n, netstat -nat | grep "TIME_WAIT" | wc -l. Enabled tcp_tw_reuse via echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse to reuse TIME_WAIT sockets. Nacos client logs showed no entries at dropout time.

Suspect 5: Nacos Server Failure

Nacos server nodes showed normal base metrics (memory, CPU, disk) — dozens of other services remained healthy. However, Nacos server logs confirmed active deregistration of the missing services. Why only random services on specific machines?

Suspect 6: Microservice Resource Overuse

Deployment scripts and load balancing were identical across machines; the same services ran fine elsewhere. Increased JVM heap per service and added stack printing. After waiting, the issue recurred with no stack traces (processes hadn’t exited).

Nacos Client Version Check

Search revealed a known issue with Nacos client 1.4.1 (the version in Spring Cloud Alibaba 2.2.1). Upgraded the client; issue persisted after days of observation.

Deep JVM & Heartbeat Investigation

Used arthas to monitor metrics — all normal. Hypothesized heartbeat thread was being killed. Examined Nacos client source (1.x heartbeat function) and used Arthas to watch heartbeat calls. When the anomaly recurred, Arthas hung with no output.

Built a custom monitor to catch dropout events in real time. During work hours, captured an event. Server basics checked out, but Arthas could not attach to the JVM process. jstat showed normal GC. jmap / jstack failed to attach; used jstack -F -l <pid> > heap.txt (forced, less info). The 10,000+ line stack trace revealed no deadlock or obvious anomaly. Meanwhile, the monitor reported the services had auto-recovered.

Attempted reproduction. On second occurrence: same pattern — Arthas attach fails → forced jstack → services recover. Hypothesis: JVM entered a "hung" state (not dead), freezing all threads including heartbeat and logging.

Root Cause Discovery

Searched "JVM hang troubleshooting" and found a reliable answer pointing to a Linux kernel bug. Compared kernel versions across machines with uname -r. The faulty machine ran an older kernel version. Filed an Alibaba Cloud ticket to upgrade the kernel.

Resolution

After kernel upgrade and reboot, the issue did not reappear over two days of observation. The root cause was a Linux kernel bug triggering JVM hangs that silently stopped heartbeats, causing Nacos to deregister services.

Conclusion

The investigation spanned over a week, demonstrating the importance of systematic elimination, low-level tooling (Arthas, jstack -F), and cross-layer thinking (application → JVM → OS kernel). Persistence and curiosity were key to uncovering an obscure kernel defect.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

NacostroubleshootingSpring Cloud AlibabaArthasheartbeatJava microservicesJVM hangLinux kernel bug
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.