How Baidu Maps Re‑engineered Its Indexing Unit for Scalable Data Storage

In this technical article the Baidu Maps team shares the refactoring and optimization of the indexing (build) unit, describing the problems they faced, the design thinking behind the solutions, and how the new architecture avoids system‑design gaps that previously required guesswork about trade‑offs.

The overall cloud storage and retrieval architecture was introduced in a previous article; the red‑outlined part of the diagram below highlights the focus of this refactor.

Problem

The existing build unit became the main bottleneck of the update pipeline. It stored incremental forward indexes locally and used a single‑threaded message sequence store to guarantee no duplicate pushes. The single‑threaded design caused message back‑log when QPS reached around 100, and scaling required costly sharding across multiple services, making roll‑backs complex. Additionally, the full forward index grew large, taking more than a day to rebuild and about eight hours to generate the inverted index, with poor data safety.

Design Goals

Decouple the storage access layer from routing logic so it only forwards messages without caring about target machines.

Separate business logic from data storage, allowing independent scaling of compute and storage resources.

Split the build unit into two services: one handling incremental updates, the other periodically constructing full inverted indexes.

Adopt a high‑performance distributed storage (Table) with strong write, scan, and multi‑version capabilities, eliminating the need for a separate incremental forward store.

Separate BS incremental data from full inverted data, allowing independent partitioning and eliminating unnecessary coupling.

New Architecture

The redesigned system uses Baidu’s internal distributed Table storage as the foundation for the forward index. Table provides high write throughput, versioned data for eventual consistency, fast scan performance, and easy horizontal scaling without business‑side involvement.

index_realtime_manager processes incremental updates, writes them to Table, and forwards them to the Redis‑based inverted index via a message system that guarantees ordering and consistency, removing any dependency on the AC router.

index_inverter periodically scans Table to build the full inverted index and pushes it to the search side. Both services can be scaled independently.

Conclusion

The new design makes incremental update scaling straightforward for both services and storage, and simplifies full‑index partitioning, resulting in a more extensible, reliable, and maintainable indexing pipeline for Baidu Maps.