Diagnosing Elasticsearch Search Slowdowns with STAROps Intelligent Assistant: Beyond Green Cluster Status
This article demonstrates how to use the STAROps Elasticsearch Intelligent Assistant to diagnose search performance degradation despite a Green cluster status, by drilling down to node-level metrics, correlating CPU usage, shard distribution, and slow logs, and establishing automated inspection tasks for ongoing monitoring.
An e-commerce product search service running on a 6-data-node Elasticsearch cluster experienced sudden latency spikes on a Friday at 20:00 Beijing time. Business monitoring showed search API P99 rising from ~200 ms to over 2 s with timeouts, yet the cluster status was Green, no unassigned shards existed, and cluster-average CPU was only ~40%.
Initial Investigation Reveals Node-Level Imbalance
Expanding metrics by node exposed the real issue: Node A sustained >95% CPU from 20:05 onward while the other five nodes stayed at 20–40%. The search thread pool had accumulated tens of thousands of rejections, but the cumulative counter could not reveal how many occurred during the incident. Hot-thread snapshots taken after the peak had already subsided showed no hot spots, failing to explain the earlier slowdown.
Setting Up the STAROps Elasticsearch Intelligent Assistant
Log in to the STAROps console and navigate to Skill Center → Official Recommendations.
Search for and select Elasticsearch Intelligent Assistant .
Add the skill to a digital employee responsible for the business, enable it, and complete authorization so the assistant can access the target instance.
Verify the setup with a familiar instance before using it for live diagnostics.
References: STAROps Skill Center [1], STAROps Console [2], Skill Center docs [3], Elasticsearch Intelligent Assistant docs [4], STAROps permission configuration [5].
First Inspection: Breaking Down Averages to Per-Node Metrics
The engineer invoked the assistant with the instance ID, region (cn-hangzhou), and the exact fault window (20:00–21:00). The assistant returned:
Per-node CPU time series for 20:00–21:00 — confirming Node A >95% from 20:05.
Cluster query QPS — roughly equal to the same period the previous week.
Index and shard distribution across nodes.
Missing at this stage: slow logs for the anomaly window and thread-pool rejection increments (only cumulative values were available). The engineer could now tell stakeholders: "Pressure is concentrated on Node A; investigating its shards and queries."
Example prompt used:
使用 Elasticsearch 智能助手检查杭州地域 cn-hangzhou 的实例 es-cn-xxxx,时间为故障当天北京时间 20:00—21:00。商品搜索变慢,集群是 Green,但节点 A 的 CPU 很高。请先做一轮巡检,重点按节点对比 CPU、查询负载和分片分布。Follow-Up Diagnosis: Correlating Multiple Evidence Lines Around Node A
Still in the same session, the engineer asked the assistant to:
Compare cluster query QPS with the previous week's same slot (no significant increase found).
Identify which shards of the product-search index reside on Node A.
Combine slow logs from the anomaly window to see if expensive queries concentrate on those shards or if heavier queries (larger scans, more aggregations) appeared.
List any data still not retrieved.
The assistant structured the findings in three layers:
Confirmed: Only Node A CPU >95% from 20:05; other nodes 20–40%. Cluster QPS similar to prior week — no obvious overall request-volume surge.
To Verify: Whether shard layout or data-access heat causes query pressure to concentrate on Node A; whether heavier queries (more complex aggregations, larger scans) are responsible. Root cause not yet determined.
Need Supplement: Slow logs mapped to nodes and shards for the anomaly window; thread-pool rejection increments over time.
The engineer then manually fetched slow logs, aligned them with shard-to-node mapping, and checked for any shard migrations during the window. This step determines whether Node A simply hosts more shards or whether seemingly balanced shard counts hide uneven access heat.
Remediation and Verification Loop
Based on the narrowed scope, the engineer chooses remediation:
If pressure correlates with shard layout or hot data: reassess shard sizing, primary/replica counts, routing, and actual load; redistribute.
If Node A remains high after redistribution: continue investigating that node specifically.
For read-only indices where segment count may hurt search efficiency: schedule a force-merge during low-traffic hours after evaluating resource headroom.
Because changes may be applied after the peak, the engineer records the adjustments and confirms service availability. The next comparable peak (next weekday 20:00–21:00) is used for verification. The engineer re-invokes the assistant in the same session:
"I have completed the remediation. Please re-inspect the same instance today 20:00–21:00 and compare with the fault day. Focus on inter-node CPU gaps, query load, and new rejection increments in the observation window. I will provide business API P99; please judge together whether recovery is achieved."
Verification checks: Node A CPU no longer significantly higher than peers; business P99 restored; timeouts eliminated. Only when these changes align with comparable business load can the engineer confidently hand over to the next shift.
Codifying the Process into a Long-Term Inspection Task
The team creates a recurring STAROps task so the digital employee automatically runs this inspection daily:
"Every day at 21:10 Beijing time, use the Elasticsearch Intelligent Assistant to inspect the product-search instance for 20:00–21:00. Compare per-node CPU and query load; if the inter-node gap exceeds the team-defined threshold for the agreed duration, continue diagnosis on related indices and shards. Notify on-call personnel for any anomaly or check failure."
Configuration details (instance, region, timezone, comparison method, threshold, duration, notification targets) must be explicitly defined — e.g., whether CPU gap is measured in percentage points or ratio. See "Creating and Planning Long-Term Missions" [6]. After the first run, the on-call engineer validates that the report shows per-node differences and that notifications work. Subsequent incidents can then be investigated starting from the automated check, avoiding reconstruction of the scene.
Other Anomaly Patterns the Same Inspection Can Catch
Yellow/Red cluster, unassigned shards : Follow-up checks — allocation explanation, node availability, allocation constraints, disk conditions. Key questions — which shards are affected? What exactly blocks allocation?
High disk watermark, index read-only : Follow-up checks — node disk usage, index blocks, shard distribution, recent growth. Key questions — what consumes space? Is growth continuing?
JVM/GC pressure, memory stress : Follow-up checks — heap usage, GC activity, query/write load in same window. Key questions — does resource pressure coincide with business failures? Where is it concentrated?
Each pattern follows the same principle: start from the automated per-node inspection, then drill down with targeted assistant queries to correlate metrics, logs, and topology.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
