Production Services Randomly Offline: Debugging a Linux Kernel Bug in Microservices

Background

The system runs on 11 Alibaba Cloud servers using Spring Cloud Alibaba with 60 microservices registered to a single Nacos cluster. Traffic flows: nginx → Spring Gateway → business microservices. Versions: Spring Boot 2.2.5.RELEASE, Spring Cloud Hoxton.SR3, Spring Cloud Alibaba 2.2.1.RELEASE, Java 1.8.

Incident

During a holiday, reports of "service not found" errors from the gateway appeared. Nacos console showed individual services missing (offline). The issue occurred every few days at random times, with different services dropping each time. Manual kill+restart stabilized services for 2–3 days.

Troubleshooting and Resolution

Suspect 1: Server Memory Exhaustion

Checked Alibaba Cloud console metrics for the faulty machine, but key metrics (memory usage, CPU usage, system load) were missing. Logged into the server and ran free -m – memory appeared normal. Filed a ticket with Alibaba Cloud; engineers fixed the console display issue, but the service dropout problem persisted.

Suspect 2: CPU Saturation

Commands felt responsive; top showed normal CPU usage.

Suspect 3: Disk Full

du -sh *

revealed ample disk space.

Suspect 4: Network Issues

Since services were being kicked offline due to missed heartbeats, network became the prime suspect. Used telnet, mtr -n, netstat -nat | grep "TIME_WAIT" | wc -l for rough checks. Enabled TIME_WAIT socket reuse via echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse. Checked Nacos client (microservice) logs – no relevant entries.

Suspect 5: Nacos Cluster Server Fault

Examined the Nacos cluster servers' basic metrics (memory, CPU, disk) – no anomalies (dozens of other services ran fine).

Reviewed Nacos server logs and found active deregistration operations. Yet some services on the same machine remained healthy; why were only a few randomly deregistered?

Suspect 6: Microservice Resource Overuse

Deployment scripts and load balancing were identical across machines, and the same services ran fine elsewhere. Increased each microservice's heap size and added stack-trace printing. Waited – the problem recurred but no stack traces were printed because the processes didn't exit.

Nacos Client Version Check

Found an article referencing a known issue in Nacos client 1.4.1 (the version pulled by Spring Cloud Hoxton.SR3). Upgraded the client, but after several days the issue persisted.

Deep JVM Investigation

Used Arthas to monitor metrics on healthy services – all normal.

Hypothesized that the heartbeat thread was being killed. Examined Nacos client source code (1.x heartbeat function) and used Arthas to watch heartbeat packets.

When the anomaly recurred, Arthas monitoring froze with no output.

Built a custom service-drop listener to catch the failure during working hours.

Finally captured an incident during business hours. Verified server basics were fine, but Arthas could not attach to the process.

Ran jstat for GC – normal.

Attempted jmap/jstack; jstack -l 25944 > heap.txt failed to attach. Used jstack -F -l 25944 > heap.txt (forced, less info). The 10,000+ line stack dump showed no deadlock or obvious anomaly.

At that moment the listener reported the services had re-registered automatically – the services self-recovered, making root-cause analysis harder.

Attempted to reproduce; when the second incident occurred, same symptoms: Arthas unable to attach → jstack forced → services recovered again.

Hypothesis: JVM was experiencing a "stop-the-world" or hang, freezing all threads including heartbeat and logging.

Searched "JVM hang troubleshooting" and found a credible answer pointing to a Linux kernel bug.

Kernel Version Comparison

Compared kernel versions ( uname -r) across the affected and healthy machines. The faulty machine ran an older kernel (e.g., 3.10.0-1160.el7.x86_64) while healthy machines ran a newer version (e.g., 3.10.0-1160.45.1.el7.x86_64). Confirmed the discrepancy and opened an Alibaba Cloud ticket.

Resolution

Upgraded the kernel on the problematic server and rebooted. After two days of observation, the random service dropout issue disappeared. The root cause was a Linux kernel bug that caused JVM pauses, leading to missed Nacos heartbeats and subsequent service deregistration.

Closing Thoughts

The issue tormented the team for over a week. The debugging process was a mix of frustration and breakthroughs. As engineers, we must maintain a curious, exploratory mindset – the learning never ends.