Skip to content
Cloud-Native Elasticsearch Observability & Diagnosis with Cloud Monitoring 2.0 & STAROps
Cloud Native 27 min read

Cloud-Native Elasticsearch Observability & Diagnosis with Cloud Monitoring 2.0 & STAROps

This article details how CloudLens for Elasticsearch leverages Cloud Monitoring 2.0 and UModel to unify metrics and logs per instance, then applies STAROps' four-round diagnostic framework—scoping, drilling down, log verification, and conditional recommendations—to transform ad-hoc troubleshooting into evidence-based, reproducible investigations.

Alibaba Cloud Native
Alibaba Cloud Native
Alibaba Cloud Native

Opening Scenario: The 20-Minute Blind Troubleshooting

On a quiet Wednesday at 10 AM, an on-call engineer receives a report: "Search is slow." The cluster status is green, node count unchanged, CPU ~40% across nodes, QPS flat. Only average latency has risen noticeably. Over 20 minutes the engineer checks shard allocation (normal), grabs hot_threads snapshots (inconclusive), restarts the busiest node (latency dips briefly then returns), and finally finds the culprit in an index's Searching slow log: a newly deployed aggregation query scanning a much larger range. Yet during post-mortem no one can distinguish which judgments were evidence-based and which were lucky guesses.

Core Problem

Elasticsearch troubleshooting difficulty stems not from insufficient metrics but from the lack of a common time window and object scope linking metrics, logs, and human reasoning. Evidence is scattered across cloud monitoring metrics, advanced metrics, slow logs, GC logs, and main logs—each with different granularities and query methods. By the time they are manually correlated, the optimal diagnostic window has often passed.

Solution Overview: CloudLens for Elasticsearch

CloudLens for Elasticsearch addresses this gap through three pillars:

Unified Access: Ingest metrics, advanced metrics, and multiple log types keyed by instance.

Entity Association via UModel: Connect the instance to its metric sets, log sets, and underlying storage (Prometheus for metrics, SLS Logstore for logs), making "which data belongs to this instance" a queryable fact.

Analysis Capabilities: Performance dashboards for trends, metric and log exploration for drill-down, and STAROps to structure the diagnostic process into four verifiable rounds.

Section 02: Access — Connecting Metrics and Logs to One Instance

2.1 Define Access Scope

In Cloud Monitoring 2.0's Access Center, select "Retrieval Analysis Service Elasticsearch Edition." Configure account, access policy, region, and entity scope (all resources, resource group, tags, or specific entities). Best practice: start with business-critical instances, then expand per team/account topology—avoid full-fleet onboarding initially.

2.2 Metrics: From Cluster Waterline to Internal State

Six metric sets are associated in the demo workspace:

Cloud Monitoring Metrics – Elasticsearch Cluster-Dimension Metrics: overall cluster status & performance changes.

Cloud Monitoring Metrics – Elasticsearch Cluster & Node-Dimension Metrics: drill down to node-level resource & performance differences.

Advanced Metrics – Elasticsearch Advanced Monitoring - Cluster-Dimension: supplement ES cluster internal runtime state.

Advanced Metrics – Elasticsearch Advanced Monitoring - Cluster & Node: deep dive into node, JVM, thread pools.

Advanced Metrics – Elasticsearch Advanced Monitoring - Cluster & Index: analyze index-level runtime metrics.

Advanced Metrics – Elasticsearch Advanced Monitoring - Cluster & Throttling: analyze admitted throttling metrics.

Configuration supports group selection and custom metrics. Choose metrics by reverse-engineering from the problem: for search latency, watch query volume, latency, node resources, GC, thread pools; for write issues, combine write throughput, queueing, and rejection signals.

2.3 Logs: Retaining the Scene That Explains Metric Changes

Five log types are associated:

Elasticsearch Main Log – view runtime events, exceptions, error context; align state changes with exception timestamps.

Elasticsearch Searching Slow Log – view logged search requests; combine with query latency & node pressure to analyze index, shard, request body.

Elasticsearch Indexing Slow Log – view logged write requests; correlate with write latency, queueing, rejections.

Elasticsearch GC Log – view garbage collection records; verify heap & GC metric changes, analyze concurrent collection behavior.

Elasticsearch Access Log – view recorded access requests; assist request-level troubleshooting per actual fields.

2.4 Post-Access Data Linkage

UModel diagram shows the ES instance connected to six metric sets (backed by Prometheus) and five log sets (backed by SLS Logstore). Each data type retains its native query engine; entity and association fields establish a shared analysis context. Post-access validation checks: instance discoverability, continuous metric updates, required log existence, and field alignment (region, instance ID, node) across metrics and logs.

Section 03: Troubleshooting — From Dashboard to Drill-Down

3.1 Locate Instance, Establish Unified Context

Entity Explorer lets you search, filter, and enter the target instance. Cross-account/region environments require confirming instance name, ID, and region to avoid mixing similarly named resources or cross-region data. Instance detail page aggregates dashboard, topology, health, log exploration, metric exploration, and associations. For performance issues, read in order: Overall Status → Query & Write → Node Resources → JVM & Thread Pools, then dive into relevant metric/log sets.

3.2 Dashboard: Identify Where Anomaly Occurs

Cluster status, node count, index/shard counts give quick overview, but green status only reflects shard allocation—not query experience. Query performance panel shows QPS, avg latency, slow query count. Observe anomaly onset; check if latency shift syncs with QPS change. Avg latency ≠ business P99; align with application metric definitions and time ranges. Node resource panel shows CPU, heap, disk, network—compare inter-node differences (cluster averages can mask single-node pressure). JVM & thread pool panel shows Old GC, query queue, rejections. Concurrent metric shifts form investigation leads; causality requires additional evidence.

3.3 Phenomenon-to-Evidence Correspondence Table

A practical lookup table maps observed phenomena to next metric/log actions and judgment points:

Latency up, QPS stable – First check metrics: query latency, slow query count; then query logs: Searching Slow Log; judgment point: change from request structure/scan range, not traffic.

Latency up, queue & rejections rise – First check metrics: query thread pool queue length, rejections; then query logs: Searching Slow Log + Main Log; judgment point: insufficient capacity or few long requests saturating pool.

Node CPU/memory disparity – First check metrics: node-dimension resource metrics (not cluster avg); then query logs: hot_threads snapshot + Main Log; judgment point: hot shard or request skew; snapshot only reflects current moment.

Old GC increment grows – First check metrics: heap water level, GC increment; then query logs: GC Log; judgment point: use increment not cumulative count to judge frequency.

Write latency high with rejections – First check metrics: write queue, rejections, refresh metrics; then query logs: Indexing Slow Log; judgment point: focus on bulk size, refresh, segment merge.

Status/node count abnormal fluctuation – First check metrics: cluster/node status metrics; then query logs: Main Log; judgment point: shard allocation, node offline events.

Left column must be confirmed phenomena ; right column gives directions to verify . Skipping left-column validation leads to the 20-minute blind troubleshooting.

3.4 Metric Exploration: From Preset Dashboard to Finer Dimensions

Dashboards suit quick scans; metric exploration enables specific follow-up questions. In instance's metric exploration menu, enter cloud monitoring or advanced metric sets. Example: advanced node metrics—view golden signals, pick specific metrics, apply filters, dimension analysis, YoY/MoM comparisons. For "slower today," pick comparable business baseline; for "single node hot," compare by node dimension—avoid treating aggregated values as node-level truth.

3.5 Log Exploration: Back to Concrete Requests & Runtime Records

From instance log exploration menu, choose main, GC, access, or slow logs. In Searching Slow Log, combine structured fields: index_name, shard_id, host to locate index/shard/node; search_time_ms for logged query latency; slow_search_log_type to distinguish phases; raw content for request structure inspection.

3.6 Association Topology: Verify Data Provenance

When "instance visible but data missing," use association topology to check entity-to-metric/log-set-to-storage links. Distinguishes "no matching records currently" from "data not yet ingested or associated."

Section 04: Diagnosis — STAROps Four-Round Verifiable Framework

With metrics and logs linked per instance, enter STAROps Intelligent Analysis from instance page. Interfaces: Smart Query, Quick Investigation, Deep Investigation. User describes problem around current instance, then follows up through analysis results. The following illustrates a reusable read-only investigation flow for "search slowed." Prompt examples show how to ask; they do not represent executed results—actual analyzable data depends on workspace ingestion and session permissions.

4.1 Round 1: Pin Down Instance, Time, Business Phenomenon

"Help me check if ES has issues" lacks scope. Effective start: provide instance, region, absolute time window, business phenomenon, and require data coverage check first.

Analyze instance {instanceID} from {startTime} to {endTime} (UTC+8) for search performance.
Business phenomenon: search request latency increased.
First confirm available cloud monitoring metrics, advanced metrics, and logs; then timeline-check query volume, query latency, CPU, heap, GC, thread pool queue & rejections.
Distinguish confirmed facts, pending hypotheses, missing data; note metric units & aggregation methods.

Fixed time window is critical. "Last hour" shifts with conversation, causing successive conclusions to draw from different data. Application-side latency definition (avg vs P99, inclusion of network/queue/client retries) must be explicit.

4.2 Round 2: Drill from Cluster to Nodes & Indices

Cloud monitoring metrics confirm anomaly onset & scope; advanced metrics analyze node differences, JVM, thread pools, index dimensions.

Compare across nodes in same time window: CPU, heap, GC increment, query thread pool queue & rejections.
State whether results are node values or cluster aggregates.
Select comparable business baseline.
If anomaly concentrates on specific nodes/indices, provide evidence; if data insufficient, state what else is needed.

QPS reflects request count; complexity depends on query structure, scan range, aggregations. Even stable QPS warrants request content & inter-node load checks. Similar shard counts ≠ similar request heat—validate with data.

4.3 Round 3: Verify Causes with Logs, Verify Caliber with Raw Metrics

If metrics point to search requests or GC, query slow logs, GC logs, main logs for the same window to map metric leads to concrete records.

Query Searching Slow Log in same window; aggregate latency distribution by index, node, query phase; give representative records.
Verify log threshold & coverage; state which phenomena have log support, which remain hypotheses.
If judging frequent GC, return raw metric meaning, window increment, and corresponding GC logs—do not treat cumulative count as per-minute rate.

Three calibers must be distinguished: cumulative count, fault-window increment, per-unit-time rate. If raw metric is cumulative since process start, current value alone cannot reveal recent minute activity; lifetime average (cumulative/uptime) cannot substitute fault-window analysis. Node restarts or counter resets require reset-aware calculations. Similarly, post-fault hot_threads is a current snapshot—cannot alone negate prior CPU peaks. Every evidence piece must declare its covered object and time range.

4.4 Round 4: Output Conditional Recommendations with Verification Plan

Structure conclusion as: Confirmed Facts / Pending Hypotheses / Missing Data / Recommended Actions.
Each recommendation: basis, preconditions, potential impact, verification method. Analysis only—no config changes.
List metrics & logs to re-check under comparable business load.

A usable diagnosis includes: confirmed facts with evidence, possible causes with validation methods, data gaps, and applicability conditions. For query rewrite, shard adjustment, or resource changes, specify impact scope & rollback. Effectiveness must be re-verified under comparable load & metric definitions—not by comparing two arbitrary-time charts (same trap as "restart caused brief latency drop").

Section 05: Integration — Bringing Observation Entry & Expertise Closer to Users

5.1 CloudLens View Embedded in Elasticsearch Console

Under "Elasticsearch Insight" in ES console: Resource Center, Insight, Alert Management, Metric Exploration, Insight Settings. Enter instance → view instance dashboard → drill into metrics/logs. Reduces steps for daily ES console users; multi-product teams can still enter via Cloud Monitoring 2.0 workspace.

5.2 STAROps Integration with Elasticsearch Intelligent Assistant

STAROps Skill Center → Official Recommendations → Elasticsearch Intelligent Assistant (skill ID: alibabacloud-elasticsearch-ops-agent). Managed ES Agent provides ES-domain analysis: instance health patrol, fault diagnosis & root cause, instance/node/index info query, network config view, knowledge Q&A. Supports multi-turn follow-up & context addition. In a digital employee with this skill, continuously question around a concrete instance. Skill boundary: read-only analysis—no auto create/delete/restart/scale; patrol/diagnosis suggestions still require evidence & operational process.

STAROps Native Intelligent Analysis – Query & investigate around workspace instances, metrics, logs; focus: pin instance & time range, check data coverage, correlate multi-source evidence.

Elasticsearch Intelligent Assistant Skill – Invoke managed ES Agent for ES-specific patrol, diagnosis, info query, knowledge Q&A; focus: define ES problem clearly, continuously ask for basis, respect read-only boundary.

Both complement each other around the same operational issue.

Conclusion

Returning to that Wednesday morning: the engineer eventually found the root cause, but post-mortem couldn't separate evidence-based judgments from luck. Had metrics and logs been pre-associated per instance, CloudLens for Elasticsearch plus STAROps would have skipped at least half the trial-and-error—not because experience is unimportant, but because evidence already waited in the same time window to be linked. CloudLens + STAROps moves "evidence gathering" from human brain to system, making every diagnostic step traceable and reproducible. Act now: in Cloud Monitoring 2.0 Access Center search "Retrieval Analysis Service Elasticsearch Edition" to start configuration.

References

Cloud Monitoring 2.0: https://cmsnext.console.aliyun.com/

Elasticsearch Insight: https://elasticsearch.console.aliyun.com/cn-shanghai/es-insight

STAROps Skill Center: https://starops.console.aliyun.com/skills/official

How to Use STAROps Elasticsearch Intelligent Assistant to See Through "Search Slowness": https://mp.weixin.qq.com/s?__biz=MzUzNzYxNjAzMg==&mid=2247588489&idx=1&sn=0a1da3f8e7fb169a3a54265735014111&scene=21#wechat_redirect

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Cloud NativeElasticsearchObservabilityTroubleshootingUModelSTAROpsCloud Monitoring 2.0CloudLens
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.