Why Time‑Series Databases Are the Backbone of IoT and Cloud‑Native Ops

IoT Growth and the Rise of Time‑Series Databases

Forrester predicts that the industrial value created by the Internet of Things will be 30 times that of the traditional Internet, and by 2020 China’s IoT market is expected to exceed five trillion yuan. New‑generation technologies such as 5G, AI and blockchain are rapidly converging with IoT, generating massive streams of sensor data that must be stored and analyzed efficiently. As the volume of time‑series data expands, a dedicated time‑series database becomes a critical "must‑answer" for enterprises.

Why High‑Performance Time‑Series Databases Matter

A time‑series database (TSDB) is vertically optimized for chronological data. For example, if a hotel has 200 rooms occupied at 8:00 PM, the value "200" recorded at that timestamp is a time‑series datum. In IoT and monitoring scenarios, metrics such as server CPU usage, electric‑vehicle telemetry, or application KPIs can reach tens of millions or even billions of points, with individual collections exceeding tens of gigabytes. These data must be written within strict time windows, requiring high concurrent write throughput and strong compression.

Beyond ingestion, TSDBs must support near‑real‑time queries for visualization and decision‑making. Traditional relational databases struggle with compression and query latency, whereas TSDBs are designed for high throughput, low latency, and high compression ratios, making them suitable for manufacturing, finance, social media, energy, smart homes, and many other domains.

Decades of Experience and Emerging Challenges

IDC forecasts that global data will reach 175 ZB by 2025, with 30 % being time‑series data. Over the past ten years, more than 30 time‑series databases have appeared. Compared with relational databases, TSDBs are simpler—lacking complex transactions and per‑row updates—but building a robust TSDB is as demanding as building a car: it requires expertise in storage, security, distributed systems, compilers, algorithms, data structures, and architecture design.

Cloud‑Native Storage‑Compute Separation: Huawei Cloud’s Practice

Time‑series data is also a core component of cloud‑provider infrastructure. Huawei Cloud’s rapid expansion created monitoring data that outgrew open‑source TSDBs, with metric counts jumping from millions to billions and write rates soaring from hundreds of millions to tens of billions of points per second. To meet this demand, the Huawei Cloud Data Innovation Lab launched the development of GaussDB (for Influx) in 2018, which entered commercial service in 2020.

GaussDB (for Influx) adopts a cloud‑native storage‑compute separation architecture, offering minute‑level elastic scaling without data migration. It supports billions of time lines, trillions of daily writes, lossless compression with a 1:20 ratio, and leverages MPP, vectorization, and pre‑aggregation to outperform open‑source solutions such as OpenTSDB and InfluxDB, especially for single‑time‑line and multi‑dimensional aggregation queries.

After migrating a workload from Cassandra to GaussDB (for Influx), compute nodes dropped from 39 to 9 (a 4‑fold reduction) and daily storage consumption fell from 1 TB to under 100 GB (a 10‑fold reduction). Today, GaussDB (for Influx) serves more than 15 internal and external customers and has become a key part of Huawei Cloud’s infrastructure.

R&D Realities: Tackling OOM and Metadata Scaling

The development journey highlighted challenges such as process OOM (out‑of‑memory) caused by massive metadata loading. In a TSDB, each data file’s metadata—time‑line counts, offsets, etc.—must be kept in memory for indexing. When time‑line numbers reach the hundred‑million level, metadata can no longer fit into memory on unchanged VM specs, leading to OOM during restart.

Resolving this required careful throttling of metadata loading, balancing memory pressure against restart latency—a decision that relies heavily on long‑term system‑development experience.

The team’s “hard‑core” attitude—confronting problems directly—mirrored a craftsman’s spirit, resulting in a robust, high‑performance TSDB that now underpins critical Huawei Cloud services.