Design and Implementation of a Large-Scale User Behavior Analytics Platform
The article outlines Meituan‑Dianping’s “Sensors Analytics” platform, a privately‑deployed, open‑PaaS solution that collects full‑stack user events from iOS, Android, Web and WeChat, maps IDs in near real‑time, stores detailed records in Kudu (real‑time) and Parquet (offline), and serves low‑latency queries via Impala, addressing the architectural and operational challenges of high‑throughput ingestion and data‑security requirements.
This article summarizes a talk from Meituan‑Dianping Tech Salon on the design and implementation of the “Sensors Analytics” large‑scale user behavior analysis platform.
The product must support private deployment to meet data security, privacy, and data‑asset accumulation requirements, which introduces significant operational and architectural challenges.
Key design principles include an open PaaS platform built on open‑source technologies, full‑stack data collection across iOS, Android, Web, and WeChat, and an ID‑Mapping service with trade‑offs for real‑time import latency.
Technical decisions: use of ROLAP storage of detailed records, employing Kudu for real‑time writes (WOS) and Parquet for offline query (ROS); data ingestion via Nginx HTTP, extraction to Kafka, and a KafkaConsumer MapReduce job that writes to Kudu and later converts finished tables to Parquet files of ~512 MB.
Query layer consists of a WebServer forwarding requests to a QueryEngine that translates them to SQL for Impala, which queries the unified view over Kudu and Parquet.
The architecture balances private deployment constraints, high‑throughput ingestion (hundreds of thousands of events per second), and low‑latency analytics.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
Meituan Technology Team
Over 10,000 engineers powering China’s leading lifestyle services e‑commerce platform. Supporting hundreds of millions of consumers, millions of merchants across 2,000+ industries. This is the public channel for the tech teams behind Meituan, Dianping, Meituan Waimai, Meituan Select, and related services.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.