How HuoLala’s “Lingmu” Platform Revolutionizes Map Service Testing with Real‑Time Traffic Replay

Background and Challenges

As HuoLala’s business grows, its map service becomes a critical backbone for order fulfillment, requiring accurate real‑time location, route planning, ETA, and road‑closure avoidance. Testing these capabilities is difficult because real‑world information is hard to obtain, experience‑based effects are not covered, and dynamic traffic, construction, and driver habits introduce uncertainty.

Real‑time physical world data is hard to acquire

Route feasibility can only be confirmed after actual travel.

Different map vendors may show conflicting road‑construction information.

ETA accuracy depends on traffic, signals, highways, closures, and driver behavior.

Experience‑related effect testing is missing

Subjective navigation quality is hard to evaluate early.

Frequent map updates may introduce regressions that are difficult to detect.

Solution and Goals

After researching common traffic‑recording and replay approaches, HuoLala built a custom platform—Lingmu—that satisfies three goals:

Establish traffic recording and replay capability with zero intrusion, support for C++ services and gRPC, and multi‑cluster replay with map‑specific diff.

Expand map‑wide testing coverage by using recorded traffic as ground truth for each product iteration.

Improve problem‑analysis efficiency by online, visual diff and analysis of route, time, and distance variations.

The Lingmu platform consists of presentation, logic, storage, and data layers, providing open APIs for traffic query and replay triggering.

Core Architecture

Traffic Recording

Recording must be zero‑intrusive and resource‑light, while supporting near‑real‑time replay. Existing dynamic code injection or offline log parsing cannot meet map‑specific needs, so Lingmu uses asynchronous traffic reporting: services write messages to Kafka, then an offline storage service persists them for later replay.

Replay Scheduling

Because map services use gRPC, a dynamic client that recompiles Proto files on the fly is employed, eliminating the need to regenerate stubs after protocol changes.

/** * Proto协议文件更新时，触发下载更新，并重新编译 */

Four replay strategies are offered (batch, silent, etc.).

Result Diff

Map results vary due to dynamic traffic, road closures, and serialization precision. Lingmu applies multi‑level noise reduction before diff to avoid false failures.

/** * 加载降噪配置，执行diff流程 */

Problem Analysis

An intelligent analysis stack (data source, replay encapsulation, analysis API) enables one‑click issue diagnosis, reducing manual effort from days to minutes.

Challenges and Solutions

Massive Traffic Selection

Elasticsearch’s deep‑paging limit (10 000) is bypassed by implementing a scrolling iterator that retrieves traffic data in batches.

/** * Index setting describing the maximum value of from + size on a query. */

Environment Noise

Differences between production and test environments introduce noise; Lingmu supports node‑specific replay to mitigate this.

Practical Impact

R&D Efficiency

Enables developers to self‑test with real traffic.

Facilitates incremental testing across clusters.

Improves test coverage of models, strategies, and data.

Reduces diff analysis time from two person‑days to minutes.

Product Experience

Higher driver satisfaction through more accurate navigation.

Rapid detection of extreme bad cases such as unexpected road closures.

Future Plans

Extend capabilities to multi‑node replay, intelligent data analysis, and core navigation issue diagnosis.

Support international logistics scenarios.

Integrate with existing CI/CD, automation, and performance platforms for maximum benefit.