How Meituan Built a Fault‑Tolerant Instant Logistics Platform at Scale

Background

Meituan Waimai has been developing for five years, and instant logistics for over three years. The business grew from zero to scale, accumulating distributed high‑concurrency system experience. Two main takeaways:

Instant logistics tolerates almost no failures or high latency; as complexity grows, the system must be distributed, scalable, and fault‑tolerant, ultimately eliminating downtime risk.

Focusing on cost, efficiency, and experience, the system heavily integrates AI for pricing, ETA, dispatch, capacity planning, subsidies, accounting, voice interaction, LBS mining, operations, and monitoring, achieving scale, experience, and cost reduction.

Key technical challenges include massive order‑rider matching, traffic spikes during holidays or bad weather, near‑zero tolerance for failures, and stringent real‑time data accuracy requirements.

Meituan Instant Logistics Architecture

The platform provides three core services: SLA fulfillment (ETA, pricing), multi‑objective rider matching, and rider decision support (voice, route recommendation, store arrival reminders).

Underlying this is a distributed system built on Meituan’s public components: front‑end traffic is load‑balanced by HLB; services communicate via OCTO (service registry, discovery, load balancing, fault tolerance, gray release) or message queues like Kafka and RabbitMQ; storage uses Zebra for distributed DB access; monitoring uses CAT; caching uses Squirrel+Cellar; scheduling uses Crane.

Challenges addressed include stateful cluster scalability, node hotspots, and resource imbalance.

Solutions: converting stateful nodes to stateless, leveraging parallel computation for rapid scaling; ensuring consistency via Databus, a high‑availability, low‑latency change‑data‑capture system that propagates DB binlog changes to ES, other DBs, and KV stores.

High availability is ensured through pre‑deployment capacity testing, periodic health checks, random fault injection, real‑time alerts, rapid fault localization, and post‑incident rollback, throttling, circuit breaking, and fallback mechanisms.

Single‑IDC Rapid Deployment & Disaster Recovery

After a single IDC failure, entrance services detect faults and automatically switch traffic; rapid IDC scaling pre‑synchronizes data and services, opening traffic once ready. All data‑sync and traffic‑distribution services must support automatic fault detection and removal, with scaling per IDC.

Multi‑Center Attempts

Meituan groups multiple IDC partitions into virtual centers; services are deployed uniformly across centers; when a center reaches capacity, new IDC is added to expand.

Unit‑Based Attempts

Unit‑based design offers superior partition disaster recovery and scaling; traffic routing is based on region or city; cross‑region data sync may incur latency; SET disaster recovery ensures rapid failover to other SETs.

Core Intelligent Logistics Technologies & Platform

The Machine Learning Platform provides end‑to‑end model training and algorithm deployment, addressing repeated development and data quality inconsistencies between online and offline.

JARVIS is an AIOps platform focused on stability, consolidating noisy alerts, automating fault analysis, and improving response speed and reliability.

Future Challenges

Future challenges include micro‑service bloat as business complexity grows, network amplification from minor latency, and rapid fault localization in complex service topologies, which AIOps must address. Additionally, moving from cluster‑level to unit‑level operations after unitization poses significant deployment challenges.