Evolution and Optimization of Tencent Music Content Library Data Platform: From Architecture 1.0 to 4.0

Tencent Music Entertainment operates a massive music content library and needed a unified data platform to provide inventory checks, audience segmentation, metric analysis, and tag selection for business empowerment.

Business Requirements : support real‑time availability, partial column updates, high performance, and cost control for over 800 tags and 1,300 metrics across 80+ source tables.

Architecture Evolution :

1.0 : Traditional three‑layer warehouse (ODS‑DWD‑DWS), ClickHouse for analytics, Elasticsearch for tag selection. Problems included delayed updates, high storage cost, and complex coupling.

2.0 : Replaced ClickHouse with Apache Doris, introduced partial‑column updates via the Aggregate model, and split wide tables into smaller partitions to reduce storage and improve throughput.

3.0 : Added a semantic layer to centralize tag and metric definitions, reducing inconsistencies and simplifying application queries.

4.0 : Deepened semantic layer integration, making it the core node that drives computation, acceleration, and query planning, achieving unified view and standardized management.

Data Model Choice : Adopted Doris' Aggregate model with REPLACE_IF_NOT_NULL for partial updates; compared Unique and Duplicate models for future use.

Write Optimization :

Generated wide tables in Flink (instead of TDW) to lower storage cost and improve latency.

Implemented batch size adaptation and BE submission balancing.

Adjusted Doris compaction parameters using max_XXXX_compaction_thread and max_cumulative_compaction_num_singleton_deltas.

Query Optimization : Combined Doris with Elasticsearch (Doris on ES) to handle tag‑based filtering and metric aggregation. Example query:

SELECT tag, agg(metric)
FROM Doris
WHERE id IN (SELECT id FROM Es WHERE tagFilter)
GROUP BY tag

Added session variable es_optimize, bucket‑join strategy, and bitmap compression to reduce network I/O and join cost, cutting million‑record query time from 60 s to 3.7 s.

Cost Optimization :

Used Doris TTL to keep only recent data, moving older data to TDW.

Adjusted replica counts by data age and employed hot‑cold tiering (SSD for 7‑day hot data, HDD for older).

Leveraged Doris' simple FE/BE architecture and MySQL compatibility to lower operational overhead.

Future Plans : Automatic hot‑cold data identification with Iceberg, materialized views for frequent tag‑metric combos, and broader use of Doris for warehouse compute tasks.

The article concludes with acknowledgments to the Apache Doris community and SelectDB for their support.