Best Practices for Full‑Stack Operations Monitoring and Cost Reduction Using Alibaba Cloud Elasticsearch

The presentation introduces a best‑practice solution for full‑stack operations monitoring and cost reduction, built on Alibaba Cloud Elasticsearch, and is organized into three parts: the current monitoring landscape, typical problems with corresponding solutions, and a concrete use‑case.

Current monitoring status – Modern operations face increasing complexity due to heterogeneous infrastructure (servers, GPUs, network, storage), multi‑level distributed services, containerization, and cloud‑native architectures such as Kubernetes. These factors make it difficult to obtain a clear, end‑to‑end view of system health and to quickly diagnose anomalies.

Trend toward intelligent operations – The evolution moves from traditional ITOM to ITOA and finally to AIOps, where large‑scale data collection, preprocessing, indexing, and machine‑learning‑driven analysis enable proactive fault prevention and system‑level intelligence.

Common problems and solutions – A robust full‑stack observability platform must answer questions about deployment architecture, resource usage, call relationships, performance, and bottlenecks. The widely adopted ELK stack (Elasticsearch, Logstash, Kibana) satisfies these needs, offering unified handling of metrics, logs, and distributed tracing.

The baseline ELK architecture consists of data collection agents (Beats, APM), Logstash for transport, Elasticsearch for storage and analysis, and Kibana for visualization. However, it suffers from high cost, write‑performance bottlenecks, query‑write interference, and limited analytics capabilities.

Optimization dimensions

Write‑path improvements – Introduce Kafka as a buffering layer and Flink for data structuring, adjust bulk write settings, increase write threads, tune refresh_interval, and modify translog to asynchronous mode. Alibaba Cloud Elasticsearch adds IndexingService for managed indexing and Fastbulk for efficient bulk handling, plus physical replication to reduce CPU load.

Storage/processing enhancements – Deploy hot‑cold node configurations (SSD for hot data, SATA or cloud disks for cold data), use ILM/DataStream for lifecycle management, and apply codec compression plugins. Alibaba Cloud offers shared hot‑cold resources, Openstore for cold data, and further compression gains.

Query performance tuning – Optimize field types (e.g., use keyword instead of numeric where appropriate), switch JVM GC from CMS to G1, add dedicated coordinating nodes, and enable async search. Alibaba Cloud provides cold‑query isolation, slow‑query pools, segment pruning for time‑series data, and GIG flow control for node selection.

Intelligent analysis (AIOps) – Leverage X‑Pack Platinum features such as machine‑learning, anomaly detection, and advanced reporting, together with Alibaba Cloud’s intelligent operations tools for automated diagnostics, capacity planning, and alerting.

Use‑case – A major automotive manufacturer migrated its fragmented, containerized services and petabyte‑scale data to the proposed architecture. By rebuilding the ingestion pipeline, adopting Alibaba Cloud Elasticsearch, and applying the optimizations above, the customer reduced overall costs by ~40%, achieved peak write throughput of 200 MB/s, and eliminated incidents for six months while enabling new business lines and security analytics.

The article concludes with a call to try Alibaba Cloud Elasticsearch and provides links to detailed documentation for each optimization technique.