Evolution of Meituan Instant Logistics Distributed System Architecture and Technical Challenges

Article #308 – 2018, Issue #100

The article is based on a talk by Meituan senior technical expert Song Bin at the ArchSummit architecture conference. It describes the progressive evolution of Meituan’s instant‑logistics distributed system, the technical obstacles encountered, and the solutions adopted.

Background

Meituan Waimai has been operating for five years, and its instant‑logistics service for more than three years. The business grew from zero to a large‑scale, high‑concurrency system, yielding two major insights:

Instant‑logistics tolerates almost no failure or latency; the system must be distributed, scalable, and fault‑tolerant.

Cost, efficiency, and user experience drive heavy integration of AI across pricing, ETA, dispatch, capacity planning, subsidies, accounting, voice interaction, LBS mining, operations, and monitoring.

The main technical challenges in the evolution are:

Massive order and rider scale leading to ultra‑large matching computations.

Holiday or severe‑weather spikes where traffic surges to dozens of times normal.

Logistics fulfillment is a critical online‑to‑offline link; failures are unacceptable.

Stringent real‑time data accuracy and latency requirements.

Meituan Instant‑Logistics Architecture

The platform focuses on three aspects: (1) SLA for users – ETA calculation and pricing; (2) Multi‑objective (cost, efficiency, experience) rider‑matching; (3) Rider‑side decision support such as smart voice, route recommendation, and store‑arrival reminders.

All of these are underpinned by a robust distributed system that guarantees high availability and high concurrency.

Distributed architecture follows the CAP theorem (Consistency, Availability, Partition tolerance). Services are deployed on multiple peer nodes that communicate over the network, forming clusters that provide consistent, highly available services.

Initially Meituan used vertical services per business domain, then introduced layered services for availability, and finally migrated to micro‑services after careful evaluation.

Distributed System Practice

The typical Meituan distributed system stack includes:

HLB for front‑end traffic load balancing.

OCTO for service registration, discovery, load balancing, fault tolerance, and gray releases.

Message queues such as Kafka or RabbitMQ for asynchronous communication.

Zebra for distributed database access.

CAT (Meituan’s open‑source distributed monitoring) for log collection and monitoring.

Squirrel+Cellar for distributed caching.

Crane for distributed task scheduling.

Key problems addressed:

Stateful clusters have poor scalability; Meituan transformed stateful nodes into stateless ones and leveraged parallel computation for rapid scaling.

Data consistency between DB and cache is solved by Databus, a high‑availability, low‑latency change‑data‑capture system that streams binlog changes to downstream stores (ES, KV systems, etc.).

High availability is ensured through full‑link stress testing, periodic health checks, random fault injection (service, machine, component), real‑time alerts (performance, business metrics, availability), fast fault localization, and post‑incident rollback, throttling, circuit‑breaking, and fallback mechanisms.

Single‑IDC Rapid Deployment & Disaster Recovery

After an IDC failure, entry services detect the fault and switch traffic automatically. Rapid IDC scaling is achieved by pre‑synchronizing data, pre‑deploying services, and opening traffic only after readiness. All data‑sync and traffic‑distribution services must support automatic fault detection and removal.

Multi‑Center Attempts

Meituan groups multiple IDC partitions into a virtual center; services are deployed uniformly across the center. When capacity is insufficient, new IDC nodes are added to expand the center.

Unit‑Based Attempts

Compared with multi‑center, unit‑based design offers finer‑grained disaster recovery and scaling. Traffic routing is performed by region or city; cross‑region data sync may incur latency, so SET disaster recovery ensures rapid failover to another SET when needed.

Core Technical Capabilities and Platform Consolidation

The Machine Learning Platform provides an end‑to‑end environment for model training and algorithm deployment, addressing the challenges of diverse algorithmic scenarios, duplicated effort, and inconsistent data quality.

JARVIS is an AIOps platform focused on stability assurance. It reduces alarm noise, automates fault detection, and accelerates root‑cause analysis for large‑scale distributed clusters.

Future Challenges

Future challenges include:

Micro‑service explosion as business complexity grows.

Network amplification effects caused by even slight latency in a mesh of services.

Rapid fault localization in complex topologies, a key AIOps problem.

Transition from cluster‑level to unit‑level operations, demanding new deployment and maintenance capabilities.

Author Biography

Song Bin, senior technical expert at Meituan, has been involved in distributed system architecture and high‑concurrency stability for years. He leads the backend of the instant‑logistics team, focusing on scheduling, settlement, LBS, pricing, algorithmic data platforms, and stability assurance.

Recruitment Information

Meituan delivery technology team is hiring senior experts and architects in LBS, scheduling, settlement, AIOps, machine‑learning platforms, and algorithm engineering. Interested candidates may send resumes to [email protected] or [email protected].

Advertisement

From Dec 7‑8, Meituan technical experts will share experiences on knowledge‑graph construction, anomaly detection, Hybrid‑App practices, and automated testing at the Beijing ArchSummit. Registration is open.

Front‑end technology producer: Feng Yong, Meituan Financial Services R&D Director.