Alibaba Cloud Elasticsearch Log Scenario Best Practices, Engine Optimizations, and Performance Evaluation

Log Scenario Overview

Alibaba Cloud Elasticsearch (ES) operates over 10,000 clusters and 60,000 nodes, with log clusters accounting for 35% of the fleet, each typically exceeding 40 CPU cores and handling petabyte‑scale data across industries such as gaming, healthcare, and automotive.

Cluster Characteristics

High write throughput: daily ingestion can reach tens of terabytes.

Second‑level query response: essential for microservice troubleshooting and monitoring dashboards.

Large storage volume: massive daily writes and long retention periods.

Preference for recent data access.

Pain Points of Self‑Built Log Clusters

High cost due to intensive compute and storage usage.

Slow scaling and recovery because of large data volumes.

Poor stability from lack of cold/hot query isolation.

Complex data lifecycle management.

Best‑Practice Optimizations (Five Areas)

Cold/Hot query shared compute resources – reduces compute cost by 50% while maintaining stability.

Low‑cost intelligent massive‑storage engine – cuts storage cost by 70%.

Compute‑storage separation – enables fast elastic recovery of massive cold data.

Automatic cold/hot data migration – eliminates manual operations.

Managed indexing service – offloads indexing work, boosting write performance over 10×.

Engine Kernel Optimizations

1. Shared cold/hot compute resources via AJDK tenant isolation ensure hot queries are prioritized.

2. Query performance improvements:

Row‑based to column‑based query transformation reduces multi‑threaded DocValue retrieval latency.

AJDK Wisp coroutine technology lowers thread‑switch overhead.

Block pre‑read strategy accelerates posting‑list access.

Codec enhancements with large‑block compression reduce I/O calls.

3. Intelligent caching (SmartCache) applies adaptive eviction policies, such as N‑LRU for DocValue indexes, preventing cache saturation during analytical queries.

Write Performance Enhancements

Indexing Service centralizes indexing, delivering >10× write throughput.

Dual‑cluster active‑passive setup ensures high availability.

Pay‑as‑you‑go managed write service further lowers costs.

Compute‑Storage Separation

Cold data resides on shared storage; writes occur only on primary shards while replicas read from the shared store, eliminating data copy during scaling or node failures.

Performance Evaluation

• Storage cost: intelligent massive‑storage engine costs ¥0.15/GB·month, 60% lower than high‑efficiency cloud disks and >80% lower than SSDs.

• Query latency: the optimized engine reduces latency by ~30% compared to high‑efficiency cloud disks for both simple time‑range queries and aggregation‑heavy analytical queries on a 1.5 TB index across 24 × 16C64GB nodes.

• Write TPS: Indexing Service clusters achieve 7–10× higher write throughput than native ES across 2C8GB, 4C16GB, and 8C32GB specifications.

Product Demonstration Steps

Access the Alibaba Cloud Elasticsearch console at https://elasticsearch.console.aliyun.com/ .

Create an Elasticsearch instance, choosing between the generic commercial edition and the log‑enhanced edition.

Configure region, zone, and select cold/hot shared resource specifications and intelligent storage options.

Set up VPC and virtual switch for network connectivity.

Specify instance name, login credentials, and confirm the order.

Access the cluster via Kibana, configure public‑access whitelist IPs, and log in.

Use Kibana’s Dev Tools to query cluster metadata, create indices, insert documents, and perform searches.

Enable advanced monitoring and alerting for cluster and node metrics.

Utilize built‑in log views (main log, search slow log, indexing slow log, GC log, ES access log) for troubleshooting.

The demonstration showcases the end‑to‑end workflow of provisioning, configuring, and operating an Alibaba Cloud Elasticsearch log cluster with the described optimizations.