OPPO's Real-Time Data Warehouse Architecture and Practices Based on Apache Flink

Speaker Zhang Jun, OPPO's big‑data platform R&D lead, introduces OPPO's real‑time data warehouse built on Apache Flink, which processes over 10 trillion records per day with peak throughput exceeding 300 million events per second.

Background : OPPO's mobile internet services (ColorOS, app store, cloud, gaming) generate massive data, requiring a data‑centric architecture centered on a data warehouse to support tagging, BI reporting, and service interfaces.

Top‑Level Design : The architecture uses open‑source components for data ingestion (Kafka), storage (ES, MySQL, Kylin, Redis, HBase), and processing (Flink SQL). Real‑time and offline warehouses share similar layers (ODS, DWD, ADS) but differ in latency requirements, aiming for minute‑ or second‑level freshness.

Implementation Highlights :

SQL development and metadata management are unified via a Flink Catalog that loads table definitions from MySQL.

Jobs are stored in a Job Store, compiled into Flink Table API graphs, and submitted to YARN.

Optimized Kafka consumption by rewriting the Stream Graph so multiple SQL statements share a single data source, reducing redundant reads.

Automated end‑to‑end data pipeline: a single SQL can generate report tables, automatically detect required dimensions/metrics, and push results to Druid and BI systems.

Full‑link latency monitoring aggregates lag across all Kafka topics and Flink tables, providing a holistic view of pipeline latency.

Multi‑tenant management includes Kafka authentication/quota and YARN queue assignment for resource isolation.

Future Outlook focuses on more convenient SQL development (enhanced expression power, additional connectors, reusable templates, coding standards), finer‑grained resource scheduling based on SQL complexity, automated parameter tuning, and self‑adaptive scaling for large‑scale clusters.

The talk concludes with thanks and invites the audience to join the DataFunTalk big‑data community.