Renrenche Mobile Data Platform: Architecture, Real‑Time Computing, and BI Solutions

The talk by Wu Shuiyong, head of Renrenche’s data platform, outlines four major parts of the platform: overall architecture, a Web IDE‑driven real‑time computation platform, an offline BI reporting platform, and mobile data‑driven practices.

Overall Architecture – The data portal comprises BI reporting, metadata management, real‑time computation, self‑service data extraction, data‑ticketing, and monitoring platforms. Data from sources such as logs, SDKs, MySQL, and Kafka are first ingested into a Hadoop cluster, forming a unified "One Data" layer that feeds a "One Service" API layer (e.g., Druid, Presto, Spark SQL, ClickHouse) and a "One Meta" metadata service for data maps, metrics, and access control.

Data Flow – Data moves from source to Hadoop, then to SQL‑on‑Hadoop for low‑latency queries, and finally to a Spark Streaming layer that processes Kafka streams for real‑time use cases like order forecasting and risk assessment. The design separates offline and real‑time pipelines to avoid the complexity of a full Lambda architecture.

Real‑Time Computing Platform – Built on Apache Spark, the platform offers a Web IDE for writing, debugging, and deploying SQL‑based streaming jobs. It supports MySQL and Kafka sources, heterogeneous joins, multi‑stream joins, custom UDFs, and three delivery guarantees (at‑least‑once, at‑most‑once, exactly‑once). Engine selection compared Storm, Spark Streaming, and Flink; Spark was chosen for its active community, stability, and SQL optimizations.

The platform’s architecture visualizes data ingestion, Kafka topic abstraction as views, and downstream outputs to MySQL, Kafka, Elasticsearch, or Druid. Data‑ticketing ensures schema governance, while the Web IDE provides project navigation, UDF management, debugging, monitoring, lineage, parameter configuration, and deployment via a YARN‑API wrapper.

BI Reporting Platform – ClickHouse was selected for its high concurrency, columnar storage, support for incremental and full sync, upserts, and sub‑second query latency. The BI layer offers a drag‑and‑drop, WYSIWYG interface for building dashboards on PC, Android, and iOS, with features such as table joins, union queries, and advanced data lineage.

Data‑Driven Operations – Over 20,000 offline staff and thousands of online users rely on the platform. Online teams use comprehensive dashboards, while offline personnel access mobile dashboards to monitor performance metrics, enabling real‑time, data‑driven decision making across the organization.

The presentation concludes with a summary of the platform’s characteristics, emphasizing low latency, high concurrency, data consistency, and the importance of aligning technical choices with business scenarios.