How One‑Click Thread Analysis Transforms Automated APM in Microservice Ops

1. Automated Operations and APM

Automated operations aim to systematize IT‑ops processes such as monitoring, execution, feedback, and optimization. The evolution includes three stages: initial tooling, process‑oriented ITIL/DevOps, and full automation where human intervention is minimized.

APM (Application Performance Management) addresses two key problems: real‑time monitoring of business, application, and infrastructure metrics, and automated problem diagnosis that captures issue snapshots and extracts root causes.

2. Why One‑Click Thread Analysis Is Needed

Traditional thread‑diagnosis workflows involve many manual steps—logging into a bastion host, running top, locating high‑CPU threads, generating a Java thread dump, and manually correlating it with code. This process is overly specialized, passive, inefficient, and cannot be parallelized, especially when multiple processes exhibit high CPU usage.

Modern automated‑ops requirements also demand unified diagnostic data storage, knowledge‑base creation, and the possibility of automated analysis using big‑data or AI techniques.

3. Architecture of One‑Click Thread Analysis

Capture

Collect raw diagnostic data from the target JVM: process resource usage (CPU, memory), per‑thread resource usage, Java thread dump (via jstack or JDK API), and business metadata (application name, business line).

Two implementation approaches are described:

Direct Dump Trigger : a capture program runs under the same user as the target process and directly invokes top, top -Hp, and jstack. Simple but limited by user permissions.

Indirect Dump Trigger : a Java agent exposes an HTTP endpoint; a remote client sends a request that triggers the same data collection. This works across user boundaries and is container‑friendly.

Aggregation

The captured dump is packaged into a message, compressed with GZip, and sent to a message queue (RocketMQ). The consumer extracts, decompresses, and forwards the data to the analysis pipeline.

Storage

Analyzed data is stored in Elasticsearch. Index management uses weekly indices (e.g., jta_2023-07-02) to keep index size manageable and enable efficient time‑range queries.

Query

Two query approaches are offered:

Direct DSL queries via Kibana/Grafana for flexible ad‑hoc analysis.

A custom query service that abstracts the DSL, provides business‑oriented templates, and simplifies integration.

Deep Analysis

Analysis templates encode expert knowledge for specific scenarios. The workflow includes defining the analysis goal, selecting relevant data, loading the template (as a JAR or plugin), executing it, and presenting results.

4. Deep‑Dive Analysis Scenarios

Scenario 1 – High CPU Analysis

Extract all runnable threads for a given PID.

Sort by CPU usage descending.

Optionally enrich with thread type (GC, middleware, user pool) to suggest root‑cause actions.

Scenario 2 – Deadlock Analysis

Beyond the built‑in JVM deadlock report, a lock‑dependency directed graph is constructed. Nodes represent threads; edges represent lock‑wait relationships. Cycles in the graph reveal deadlock rings, and the algorithm extracts involved object IDs, types, and stack traces.

Scenario 3 – Single‑Thread Wait Analysis

For a blocked thread, the template extracts the waiting lock, finds the owning thread, and determines whether the wait is due to a simple block, a chain of locks, or a deadlock.

Scenario 4 – Single‑Thread State Timeline

Collect N successive dumps of a specific thread, build per‑dump lock‑dependency graphs, and merge them into a temporal graph to visualize state transitions and pinpoint when a thread entered a waiting state.

Scenario 5 – Multi‑Thread State Timeline

Extend the temporal graph to multiple threads, revealing interactions such as cascading waits and compound deadlocks across the system.

5. Summary

The presentation covered four parts: (1) the definition of APM within automated ops and its two core challenges, (2) a classic manual thread‑analysis case that motivates one‑click automation, (3) a five‑stage architecture for one‑click thread analysis, and (4) concrete analysis templates for high‑CPU, deadlock, single‑thread, and multi‑thread scenarios.