Time Series Data Platform: Business Scenarios, Architecture, and Core Technologies
This article introduces the main business scenarios, system architecture, and core technologies of a time‑series data platform, covering data collection, processing, storage, analysis, and the specific features required for high‑performance, scalable, and reliable time‑series data management.
The talk, presented by NetEase big‑data expert Fan Xinxin, outlines the business scenarios, architecture, and core technologies of a time‑series data platform.
Business Scenarios : The platform supports system monitoring (servers, containers, services), task monitoring, application performance monitoring, link tracing, and business statistics for e‑commerce and advertising, all requiring time‑stamped metrics.
Architecture Overview : Data originates from MySQL, logs, app data, and sensors, passes through a collection layer (Sqoop, DataStream, SDK, Gateway), is ingested into Kafka, processed by Flink or Spark Streaming, and stored in offline (HDFS, Kudu, GP, Hive, Spark, Impala) and online (HBase) layers, as well as dedicated time‑series stores such as OpenTSDB, Druid, and InfluxDB.
The platform’s core is a distributed time‑series database designed by NetEase, combining advantages of Druid, OpenTSDB, and InfluxDB, offering high‑performance writes, multi‑dimensional queries, and efficient aggregation.
Data Storage Model : StorePolicy defines TTL, replica count, and shard interval. Each StorePolicy creates ShardGroups; hot data is written to ShardGroups, which are further split into Shards distributed across nodes. Shards are replicated for reliability.
Key Storage Techniques :
Column‑oriented storage of time series (high compression, efficient aggregation).
Inverted index for fast conditional queries.
Automatic shard expansion and balanced distribution across nodes.
Multi‑level storage optimization: hot shards stored on SSD, cold shards on HDD, with selective index loading.
These techniques enable the platform to meet core time‑series requirements: time‑range queries, multi‑dimensional filtering, TTL support, high compression (>10x), efficient aggregation, horizontal scalability, high availability, and data reliability.
The presenter, Fan Xinxin, is a NetEase big‑data technology expert focusing on HBase, TSDB, and industrial IoT platform construction.
DataFunTalk
Dedicated to sharing and discussing big data and AI technology applications, aiming to empower a million data scientists. Regularly hosts live tech talks and curates articles on big data, recommendation/search algorithms, advertising algorithms, NLP, intelligent risk control, autonomous driving, and machine learning/deep learning.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.