Evolution and Architecture of Beike Real-Time Computing Platform
Beike's real-time computing platform, led by Liu Liyun, has evolved from early Spark Streaming to a Flink-based system with SQL 1.0, 2.0, and upcoming 3.0, supporting a large-scale data warehouse, event-driven processing, extensive monitoring, and diverse business scenarios across the company's operations.
The presentation outlines the evolution of Beike's real-time computing platform, initially built on Spark Streaming in 2018 to address fragmented, siloed development across business lines, and later upgraded to a unified Flink-based system with standardized SQL support.
In 2019 the platform introduced Flink 1.8 with SQL 2.0, enabling real-time data warehouse capabilities; by early 2023 it began developing a unified SQL 3.0 on Flink 1.11 to improve event processing and reduce existing limitations.
The platform now serves most of Beike's business units, handling over 800 tasks and processing up to 2.5 trillion messages daily, with peak per‑task throughput of 3 million messages. Its architecture includes a compute layer (Flink, Spark), storage layer (ClickHouse, MySQL, Hive, Redis, HBase, Doris), and a management layer for task development, monitoring, and resource allocation.
Monitoring and alerting are integrated via SDKs, Java agents, and InfluxDB, providing latency, heartbeat, and data‑lineage metrics. The system also offers a visual SQL editor, debugging tools (manual and automatic sample data), and supports CEP for complex event patterns.
Event‑driven processing is addressed through a dedicated platform that abstracts event management, rule engines, and action triggers, supporting use cases such as risk control, real‑time rewards, and recommendation. It provides unified event definitions, Kafka‑based sources, CEP patterns, and configurable downstream actions (message sending, service calls, Kafka sinks).
Future plans include a user data platform for historical behavior queries, enhanced state management and recovery, dynamic resource allocation, high‑availability improvements, and hybrid batch‑stream querying leveraging data lake capabilities.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
DataFunTalk
Dedicated to sharing and discussing big data and AI technology applications, aiming to empower a million data scientists. Regularly hosts live tech talks and curates articles on big data, recommendation/search algorithms, advertising algorithms, NLP, intelligent risk control, autonomous driving, and machine learning/deep learning.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.