Why Time‑Series Databases Are Essential for Modern Monitoring: Fundamentals and Live‑Streaming Use Cases

1. Introduction to Time‑Series Databases

Time‑series databases (TSDBs) are optimized for storing and retrieving data indexed by timestamps, making them ideal for continuous, high‑frequency data such as sensor readings, logs, and monitoring metrics. Typical characteristics of time‑series data include rapid generation (multiple points per second per source), strict time ordering, and high dimensionality (many tags per metric).

Compared with relational databases, TSDBs achieve roughly half the storage space and dramatically faster query performance for time‑based queries, because they use specialized storage layouts and avoid complex schema designs.

2. Core Concepts and Data Model

A TSDB data point consists of four parts:

Metric – the name of the measured item (e.g., usercount).

Tags – key‑value pairs that add dimensions (e.g., platform=dbaplus, speaker=zhangguanshi).

Timestamp – Unix epoch time indicating when the measurement was taken.

Value – the numeric measurement (e.g., 500).

Example data point:

usercount platform=dbaplus speaker=zhangguanshi 1497344217 value=500

This structure enables efficient insertion (mostly write‑only) and time‑range queries such as latest value, statistical aggregates, and anomaly detection.

3. Industry Adoption of TSDBs

Major companies and open‑source projects use TSDBs for monitoring:

Google – Prometheus (inspired by Borgmon).

Facebook – Beringei.

Alibaba – GoldEye.

Baidu – TSDB for IoT monitoring.

Open‑source options – OpenTSDB, InfluxDB, Graphite, Druid, TimescaleDB.

The DB‑Engines ranking (see image) shows the popularity of these systems.

4. OpenTSDB Architecture

OpenTSDB is a Java‑based TSDB that stores data in HBase. Its architecture includes several HBase tables:

Data Table ( tsdb) – stores raw data points.

UID Table ( tsdb-uid) – maps string tags to numeric UIDs.

Meta Table – indexes time‑series metadata.

Tree Table – auxiliary metadata storage.

Rollup Table – stores pre‑aggregated data.

The rowkey design is a key highlight:

Rowkey: metric + timestamp + tagk1 + tagv1 … + tagkN + tagvN

Data can be ingested via simple telnet commands, for example:

$ telnet localhost 4242
put sys.cpu.user 1497344217 23 host=web01 user=mirzhang

5. Visualization and Alerting

Grafana is the most common UI for visualizing OpenTSDB data. For alerting, Bosun (a Golang‑based system from Stack Exchange) can query TSDB, apply expression‑based rules, and send notifications via WeChat, email, etc.

6. Tiger Live Monitoring Practice

Live‑streaming services generate massive, multi‑dimensional metrics (e.g., video quality, latency, user counts). Tiger Live’s monitoring stack consists of:

Data collection – UDP/HTTP/Telnet agents gather metrics from every service component.

Pre‑processing micro‑services – filter, aggregate, and enrich raw data before storage.

Storage – OpenTSDB on HBase for long‑term, high‑throughput writes.

Visualization – Grafana dashboards display real‑time health.

Alerting – Bosun evaluates custom rules and notifies operators.

Key video quality indicators tracked include perceptual stall ratio, network stall ratio, error rate, slow‑start ratio, load success rate, playback latency, and RTT. Micro‑service metrics (success rate, call latency, etc.) are also stored in the TSDB.

Challenges encountered: distributed write performance bottlenecks and uneven hash distribution of rowkeys in OpenTSDB, which required careful tag design.

7. Additional Uses

Beyond monitoring, the TSDB API is used to generate periodic reports (daily, weekly) and to feed other analytical pipelines.

8. Q&A Highlights

TSDBs are indeed the backbone of Tiger Live’s monitoring system.

They are suitable for both business‑level and infrastructure‑level monitoring.

Data ingestion is performed by Java micro‑services that forward processed metrics to OpenTSDB.

Compared with generic real‑time analytics, TSDBs offer lower storage cost for ordered time data.

Primary bottlenecks are write scalability and query performance, especially ensuring uniform rowkey distribution.