How Alibaba Cloud’s Serverless Elasticsearch Powers Data‑Driven Operations

Elasticsearch is an open‑source, real‑time distributed search and analytics engine built on Lucene, offering RESTful APIs for fast storage, query, and analysis of massive datasets, and serving as the foundation for complex query features in enterprise applications.

In 2017, Alibaba Cloud partnered with Elastic to launch Alibaba Cloud Elasticsearch, a fully compatible, pay‑as‑you‑go service that combines Elastic’s search technology with Alibaba’s cloud‑native high‑performance kernel, DAMO‑Lab NLP tokenization, and vector retrieval capabilities, improving application performance, agility, and intelligence while reducing costs.

Following the philosophy “originating from open source, beyond open source,” Alibaba Cloud Elasticsearch continuously upgrades its cloud‑native engine for observability, becoming the industry’s first Serverless Elasticsearch service. Its serverless log enhancement engine provides write‑accelerated indexing, Openstore massive storage, and read‑write separation, delivering lower cost, higher performance, and simpler usage for full‑stack observability.

The ELK stack—Elasticsearch, Logstash, Beats, and Kibana—forms a comprehensive ecosystem for log processing, full‑text search, and data analysis. Alibaba Cloud has migrated the entire ELK suite to the cloud, enabling real‑time log handling, search, and analytics, and SREWorks builds on this ecosystem to deliver best‑practice observability solutions.

1. What Is a Data‑Driven Operations System

A data‑driven operations system collects and unifies all operational data, deeply mines its value, and drives decisions through data, enabling quantitative management of production systems. It establishes a standardized operations data warehouse, defines data models, and provides services for data collection, storage, computation, and analysis.

2. SREWorks Data‑Driven Operations Platform

The platform consists of a core operations data warehouse offering a standard data model, and multiple data services that handle collection, storage, computation, and analysis, supporting quantitative operations.

Operations Data Warehouse

Built on open‑source Elasticsearch, the warehouse abstracts three major data themes and nine domains, includes cloud‑native operational entities and models, and supports flexible user‑defined entities and models.

The warehouse leverages Elasticsearch’s distributed indexing, multi‑replica, lifecycle management, and hot‑cold storage separation to ensure stability and dynamism, and integrates with tools like Logstash, Spark, Flink, and APM Server.

Data Collection Service

Integrates agents such as metricbeat, filebeat, and skywalking for managed observability data collection, covering resource metrics, events, logs, and tracing. Supports auto‑discovery via tags and custom scripts for job platform collection.

Data Compute/Analysis Service

Based on Apache Flink, it provides a one‑stop real‑time big data analysis platform with SQL support and built‑in UDFs for threshold detection, aggregation, and down‑sampling, supporting streaming processing of time‑series data.

Job platform services cover inspection, analysis, and diagnosis scenarios with customizable processing logic.

Metric Service

Offers metric definition and instance management, with built‑in basic resource and performance metrics, and supports custom metrics linked to collection services, pushing data to Kafka for downstream consumption.

Dataset Service

Provides rapid API generation from data warehouse models or user‑defined tables without coding, currently supporting Elasticsearch and MySQL data sources.

3. Data‑Driven Operations Practices with SREWorks

3.1 Stability Construction

Full‑Stack Service Observability

Increasing system complexity and cloud‑native adoption make rapid troubleshooting difficult. Leveraging metrics, logs, and tracing enables white‑box monitoring.

SREWorks integrates metricbeat, filebeat, skywalking, and custom job agents to provide managed, one‑stop observability data collection, shortening mean time to recovery.

Health Management Service

Combines Alibaba Emon’s intelligent detection with custom job services to deliver event collection, risk inspection, alarm detection, anomaly diagnosis, and self‑healing, reducing system risks and improving response efficiency.

3.2 Cost Construction

SREWorks includes a comprehensive cost management solution that visualizes resource consumption, provides detailed usage analysis, and supports cost optimization and governance at the resource level (CPU, memory, storage). Daily application‑level cost aggregation is stored in the data warehouse.

3.3 Efficiency Construction

Operational efficiency reflects platform automation and human‑to‑output ratio, measuring the value of labor resources.

3.4 Operations Center

The Operations Center provides dashboards on quality, cost, and efficiency, offering real‑time health scores, instance statistics, availability, cost proportion, resource allocation, operational efficiency, and activity statistics to support stability, budgeting, and scaling decisions.

4. Conclusion

With the open‑source release of Alibaba Cloud’s SREWorks platform, the data‑driven operations concepts and practices are shared to inspire new ideas and approaches in cloud‑native observability and operational excellence.