Elasticsearch Time Series Engine: Practices, Challenges, and Alibaba Cloud TimeStream

Wei Zijun, an Elasticsearch core expert from Alibaba Cloud, shares the topic of metric observability based on Elasticsearch, outlining why Elasticsearch is being extended to a time series engine, the challenges involved, and the features of the solution.

Why Elasticsearch becomes a time series engine

Elasticsearch was motivated to address metric storage pain points identified during community discussions, leading to a joint project to build a time series engine.

Two opposing user opinions are highlighted: one questioning the feasibility of using Elasticsearch for time series, and another asserting that it already can, prompting an investigation into the associated problems.

Typical time series scenarios

Monitoring – a core observability use case for Elasticsearch.

IoT – massive devices generate continuous metrics requiring a time series database.

Key characteristics of time series data include sampled data, a fixed data model (time, dimension, metric), a timeline concept, stable data flow without peaks, and a write‑heavy, read‑light pattern.

Requirements for a time series engine

High‑TPS writes to handle massive continuous data.

Multi‑dimensional aggregation queries rather than point lookups.

Cost‑effective storage through compression and metadata reduction.

Challenges when using Elasticsearch for time series

High learning curve and complex DSL for queries.

Slow query performance, especially with complex aggregations.

Significant storage overhead – Elasticsearch can require up to 74× the storage of InfluxDB for the same data.

Elasticsearch time series engine features

Columnar storage to reduce I/O and improve compression.

LSM‑like data structures for high‑throughput writes.

Built‑in aggregation capabilities for analytical queries.

Alibaba Cloud Elasticsearch TimeStream

TimeStream is Alibaba Cloud's self‑developed time series engine built on Elasticsearch, aiming to lower usage barriers, reduce costs, and enable broader integration.

It introduces a dedicated time series model (labels, metrics, timestamp) and supports native Elasticsearch APIs for read/write operations.

Key optimizations include:

Index settings index.time_series.start_time and index.time_series.end_time to enforce time‑bounded writes.

TimeSeries‑specific mapping fields time_series_dimension and time_series_metric that generate internal _tsid for efficient sorting.

Source suppression ( _source ) to save metadata space.

TimeStream also provides a Time Series DataStream (TSDS) that partitions data by time, enabling seamless rollovers and efficient time‑range queries.

Cost‑reduction and performance improvements

Data compression and removal of metadata fields dramatically shrink storage from >1900 MB to ~200 MB.

Built‑in downsampling creates coarser‑grained indices (1 min, 10 min, 60 min) that accelerate queries and further reduce storage.

Benchmark results show downsampled queries outperform raw indices and can be comparable or better than ClickHouse and InfluxDB.

Integration capabilities

TimeStream supports Prometheus remote‑write and Grafana integration, allowing Prometheus data to be stored in Elasticsearch and visualized via Grafana with full PromQL compatibility.

Overall, Elasticsearch equipped with TimeStream becomes a capable, cost‑effective time series engine suitable for monitoring, IoT, and observability workloads.