Adaptive Management of Elasticsearch Cluster Capacity

Overview

Elasticsearch is widely used at eBay for log analysis, data retrieval, and offline analytics. Managing dozens of clusters and tens of thousands of Kubernetes pods revealed three recurring problems: uniform resource ratios causing hot and cold nodes, traffic spikes leading to read/write failures, and idle resources limiting customer growth.

Elastic Compute and Capacity Management

High‑availability services must quickly detect performance degradation caused by traffic changes and scale accordingly. Elastic compute, first introduced by Amazon EC2, has become a core capability of cloud platforms such as Microsoft, Alibaba, and Elastic. A complete elastic‑compute system should be simple to control, provide second‑level scaling, and allocate resources on demand.

Problems in Pronto Multi‑Cluster Mode

Pronto originally provided a unified Elasticsearch service without isolated access or interface isolation. While a single‑cluster approach simplifies configuration and reduces errors, it also forces diverse workloads to share the same hardware, leading to resource contention, index‑resource sharing risks, and high operational cost as the number of clusters grows.

To mitigate these issues, Pronto switched to independent singleton deployments for each customer, giving full Elasticsearch/Kibana functionality (except X‑Pack). This eliminated cross‑workload competition but introduced new challenges: inconsistent field mappings, unbalanced shard sizes, inefficient queries, and excessive index/ shard counts that waste CPU, memory, and disk.

Adaptive Capacity Management Strategy

Pronto’s capacity management consists of the following components:

Pre‑Check : Identify the use‑case (Logging, Searching, Data‑Analysis) and adjust index settings (e.g., IndexBuffer, Translog, RefreshInterval) for Logging workloads.

Analysis : Collect peak values of CPU, MEM, DISK, throughput, and shard count over a monitoring period. Compute the minimum required node count using the formula<br/>

required_nodes = usage_rate * suggested_rate / suggested_rate

. For each use‑case, determine the dominant resource (DISK for Logging, CPU for Searching, CPU/MEM for Data‑Analysis) and calculate the needed Data and Client nodes.

Throttling : Alert on extreme metric spikes (e.g., Node API latency >1500 ms, excessive Rejections) and prevent scaling actions during two consecutive traffic cycles.

Scaling :

Scale‑up : Allocate nodes based on the analysis result, respecting the customer’s approved quota and any pre‑planned traffic schedule.

Scale‑down : Remove idle nodes only after confirming no hot‑node issues, limiting removal to ten nodes per batch, and monitoring latency and rejection metrics.

Metrics and Thresholds

Key monitoring metrics include MEM (heap usage split between JVM and OS caches), CPU (CGroup usage derived from /_nodes/stats), DISK, throughput, response time, thread pool size, and per‑node shard count. Additional safety indicators are Rejections, API Node Latency, and shard count per node (<200).

Recommended usage ranges are:

CPU: 5%‑40% (ideal ~20%)

HEAP: 60%‑90% (ideal ~80%)

DISK: 40%‑65% (ideal ~55%)

Throughput: 8‑20 MB/s (ideal ~8 MB/s)

Write/Search Rejections: 0

API Node Latency: <1500 ms

Shard count per node: <600

Node Types

Master Node – manages cluster metadata; only CPU and MEM are allocated.

Client Node – forwards read requests and supports Kibana; only CPU and MEM are allocated.

Data (Hot/Warm) Node – stores data and handles indexing/search; CPU, MEM, and DISK are allocated.

Strategy Formulation

Accurate identification of workload types drives resource allocation. For example, Logging workloads need more disk, Searching workloads need more CPU, and Data‑Analysis workloads need both CPU and MEM. Pronto’s capacity management components are illustrated in the diagram below.

Autoscaling Workflow

Autoscaling runs on top of the internal monitoring platform Sherlock. The Diagnose module scans cluster metrics, the Index Management Tool enriches data per index, and the Throttling module generates alerts. After manual approval, ScalingUp or ScalingDown actions are executed. The workflow includes pre‑check, throttling, metrics analysis, quota preparation, approval request, scaling, and notification steps.

Best Practices in Pronto

Key practices include:

Maintain separate singleton clusters per customer to avoid resource contention.

Continuously monitor MEM, CPU, DISK, throughput, and latency.

Apply use‑case‑specific index settings.

Enforce strict thresholds and safety checks before scaling.

Require manager and customer approval for any scaling operation.

Work Summary and Future Improvements

While Autoscaling has significantly reduced resource waste and improved efficiency, several areas need enhancement:

Introduce time‑series forecasting models to improve traffic prediction.

Address metric phase‑shifts across different indicator types.

Refine capacity calculations for Searching and Data‑Analysis workloads based on concrete user behavior.

Adopt incremental metric storage to speed up analysis and scaling decisions.

Despite these challenges, the system safely controls capacity while maintaining cluster availability, and ongoing iterations are expected to close the gap toward a fully elastic compute model.