How JD Finance Achieves Real-Time Capacity Assessment and Smart Alerting

Background

With the rapid growth of JD Finance’s business, the number of services and their dependencies increase daily, making it difficult to monitor system capacity, pinpoint fault impact, break down transaction latency, and identify bottlenecks in real time.

1. Intelligent Capacity Assessment

Offline load testing : Traffic from production is replayed to a test server using tools such as tcpcopy. When the test server reaches a bottleneck, the maximum QPS is recorded and converted to an online capacity estimate using a scaling factor.

Online load testing : Instead of affecting production, the weight of a single server in the load balancer is increased (Weighted Round Robin). The server’s traffic grows until it hits a performance limit, revealing the bottleneck (CPU, memory, bandwidth, or QPS fluctuations). Example Nginx configuration:

http {
    upstream cluster {
        server 192.168.0.2 weight=5;
        server 192.168.0.3 weight=1;
        server 192.168.0.4 weight=1;
    }
}

Capacity calculation : Offline/online tests give overall capacity, but not per‑method details. By treating the application as a black box and measuring:

Average QPS and latency of a method

Breakdown of I/O time (DB, RPC, disk, etc.)

Resource limits (DB connection pool size, thread‑pool size)

Example: a method with 200 QPS, 100 ms latency, 6 DB calls (10 ms each) and 40 ms business logic. With a DB pool of 30 connections, the DB‑limited QPS is 30 × 1000 / 60 = 500. Business‑logic‑limited QPS is 50 × 1000 / 40 = 1250, so the bottleneck is the database (500 QPS). After optimizing DB latency to 5 ms and reducing calls to 4, DB‑limited QPS rises to 1500, shifting the bottleneck to business logic (1250 QPS).

Business‑logic time classification :

RUNNABLE (actual CPU execution)

BLOCKED / WAITING / TIMED_WAITING (waiting for I/O)

CPU usage can be obtained from /proc or JMX. If RUNNABLE : WAITING is 1 : 1 and CPU usage is 20 %, the CPU‑limited QPS is 200 × 100 % / 20 % = 1000. Similar calculations apply to network bandwidth and other resources.

Real‑time capacity visualization (image omitted for brevity).

2. Intelligent Alerting

Root‑cause alert analysis combines topology, call‑chain data, time correlation, weight scoring, and machine‑learning models to filter alarms and quickly locate the source of a problem, dramatically reducing MTTR.

Alert processing steps :

Filter out irrelevant or duplicate alerts.

Generate derived alerts based on root‑cause relationships.

Associate alerts occurring in the same time window.

Calculate weights for each alert type.

Select the highest‑weight derived alert as the root cause.

Merge identical root‑cause alerts from different sources.

Match the root cause with historical knowledge‑base solutions and suggest remediation.

Example: RPC chain D→C→B→A, where A’s database timeout triggers cascaded alerts in B, C, D. Root‑cause analysis converges the alerts to A’s DB exception, allowing rapid resolution.

Strong association analysis uses known relationships such as call chains, DB‑app links, network‑device links, and host‑VM mappings. If multiple related devices alarm within the same window, they are considered associated, and the downstream component with the highest weight is treated as the root cause.

Machine‑learning root‑cause analysis :

Association rule algorithms (Apriori, FPGrowth) discover frequent co‑occurring alerts within time windows and generate root‑cause candidates.

Neural‑network algorithms : Recurrent Neural Networks (RNN) and Long Short‑Term Memory (LSTM) models capture temporal dependencies among alerts. Historical derived alerts serve as input, and the model predicts the root‑cause type for new alerts.

In financial systems, alert randomness and limited training data make strong‑association analysis often more effective, but ML can complement it when relationships are unknown.

Conclusion

Smart operations, driven by AI, are reshaping capacity planning and fault detection. While no single solution fits all scenarios, combining precise load‑testing methods, detailed capacity modeling, and both rule‑based and ML‑enhanced alert analysis provides a practical path toward real‑time, data‑driven operations management.