How Elasticsearch Serverless Cuts Log Analysis Costs by 50%

At the 2024 Cloud Xi Conference, Alibaba Cloud senior technical expert Jia Xinyu presented the key technologies, advantages, and value of Elasticsearch Serverless for intelligent log analysis.

Core Pain Points of Log Analysis

High resource cost : Storing massive data and handling spikes requires reserving large resources, driving up costs.

High O&M cost : Managing shards, replicas, and rollover strategies at TB‑PB scale is complex and labor‑intensive.

Read/write performance challenges : Maintaining performance under strict cost constraints is difficult.

Stability issues : Open‑source ES lacks circuit‑breakers; high load can cause abrupt performance drops and cluster crashes.

Serverless Log Analysis Capabilities

Out‑of‑the‑box, open‑source compatible : APIs and SDKs match open‑source ES, requiring no code changes.

High performance, low cost : Customized optimizations double write throughput and halve storage cost.

True pay‑as‑you‑go : Billing based on CPU usage (CU) rather than traffic, saving up to 50%.

Intelligent scheduling, no O&M : Automatic scaling and configuration tuning eliminate manual shard and replica management.

Key Technical Insights

Overall Architecture

The system separates data plane and control plane. The service layer routes traffic to compute services, while a shared distributed storage backs the data. The control plane handles application lifecycle, index metadata, quota, and intelligent operations.

Performance Optimizations

Engine architecture with read/write separation and OpenStore storage‑compute separation reduces I/O and eliminates real‑time replica sync.

ARM‑based “Yitian” servers raise safe utilization from 30% to 50% and improve resource efficiency by 66%.

Write‑path enhancements: source reuse via DocValue, custom LineAnalyzer tokenizer (+20% single‑core performance), ZSTD compression with directed routing, and storing only DocValue for keyword/numeric fields with Bloom filter pruning, achieving 300% write speed increase and 70% storage reduction.

Query optimizations: concurrent queries and in‑memory caching boost OSS query speed by 200%; shard‑level pruning and adaptive slow‑query downgrade improve latency for billions of records.

True Pay‑as‑You‑Go Billing

CPU usage (CU) is measured per request, normalized across CPU models, and only user‑consumed resources are billed, reducing costs by ~20% compared with traditional CU accounting.

Intelligent Scheduling and Auto‑Ops

Resource scheduling balances shard CU across nodes; configuration tuning monitors CU, memory, I/O, and network metrics to automatically adjust cluster and index settings.

Quick Start

Open the console and create an application.

Fill in the basic information.

Wait 1–2 minutes for provisioning.

These capabilities enable cost‑effective, high‑performance log analysis without operational overhead.