Practical Guide to Elasticsearch Monitoring and Operations

Elasticsearch (ES) is a distributed full‑text search engine offering high availability, scalability, and near‑real‑time search capabilities, widely used for data storage, search, and real‑time analytics; many services heavily depend on its availability, making ES health monitoring critical for overall service reliability.

The article introduces a series of ES operational guides, starting with an overview of monitoring tools and metrics, and presents the first installment focusing on practical ES monitoring.

Monitoring Overview : The goal is to enable daily service inspections, rapid fault detection, and swift root‑cause analysis through selected metrics.

Tool Selection :

X‑Pack + Kibana – official plugin providing monitoring dashboards; requires installation before cluster launch.

Jmxtrans‑ES + InfluxDB – custom tool that gathers core JMX data via HTTP and stores it in InfluxDB, with Grafana dashboards for visualization.

Elasticsearch‑HQ – lightweight alternative to the Head plugin, offering management UI and command‑line utilities.

Alerts are integrated with JD Cloud’s internal alert platform.

Metric Selection :

Black‑box monitoring includes cluster functionality (index creation/deletion, search rates, pending tasks), overall cluster health (green/yellow/red states), active shard percentage, and pending task counts.

White‑box monitoring covers capacity (total/used storage, disk usage thresholds), node resources (CPU, load, disk, JVM), shard count limits, thread‑pool queue lengths, traffic (index/search rates, network I/O), latency (search/index latency, slow queries), and errors (node failures, rejected requests, master logs).

Issues Detected via Monitoring :

Scenario 1: ES API timeouts causing monitoring tool failures – detected by functional monitoring timeout alerts; recommendation: perform regular inspections and ensure error‑log monitoring for rapid diagnosis.

Scenario 2: Data node disk failures leading to index creation failures – detected by functional and pending‑task alerts; recommendation: use RAID5/RAID10 and avoid sharing data directories.

Scenario 3: Excessive type creation in an index causing cluster instability – detected by pending‑task alerts and abnormal write rates; recommendation: regularly audit index write patterns and conduct service inspections.

Additional resources include links to open‑source projects such as ElasticHQ (https://github.com/ElasticHQ/elasticsearch-HQ) and custom monitoring scripts (https://github.com/cloud-op/monitor).