How Baidu Maps Powers Its Open Platform with Big Data Architecture

As the "big data" buzz intensifies, Baidu Maps leverages its massive daily data—hundreds of billions of location points and client accesses—to continuously improve navigation, routing, and platform services.

Data Exchange

Data collection is divided into real‑time and offline (T+1) based on timeliness, with logs and DB data as common sources. Log collection reuses internal components similar to Flume and Kafka, and a custom Flume+Kafka pipeline is built for fast‑growing services.

For DB data, a solution that streams changes (e.g., binlog parsing) to downstream queues is used, currently employing a company‑wide databus and the open‑source Canal, with plans to consolidate the architecture.

Offline Computing

The destination of data exchange is a Hadoop cluster, where multi‑dimensional, multi‑level calculations and storage build the open platform data warehouse. The warehouse design draws on experiences from banking and telecom to create a stable, evolvable data model, though details are omitted.

Key components include a task scheduler handling tens of thousands of daily jobs, a resource manager optimizing task execution, a metadata system serving as the meta‑layer for all services, and a quality‑monitoring system that automatically detects and resolves data anomalies.

Storage separates hot data for daily computation from cold data retained permanently on a dedicated cold‑backup cluster.

Real‑Time Computing

Real‑time scenarios such as the map “heat‑map” product analyze billions of positioning requests instantly. The architecture resembles Storm, using modules like importer, task processor, and exporter to perform complex stream processing.

Query Engine

After processing, data is exposed to users through various mechanisms. Report data is stored in MySQL for its small size and developer‑friendly SQL. User‑facing queries use Redis for high‑performance key‑value access. Internal analytics rely on a multi‑dimensional engine built on Apache Kylin.

Future work includes developing an MPP‑based ad‑hoc query capability integrated with metadata to offer more flexible analysis.

Data Products

The big‑data stack supports products such as reporting platforms, multi‑dimensional analysis platforms, heat‑maps, and B‑side web applications, typically built on a LAMP architecture. An open‑source visualization library (mapv) is also provided.

Conclusion

After more than three years, Baidu Maps’ open platform has formed a relatively complete big‑data technology stack and team. The platform continues to evolve with business needs, and the team remains committed to learning and openness.