How Kuaidi Dache Scaled to Millions: Lessons from LBS, Long Connections, and Real‑Time Data Architecture

LBS Bottleneck and Solutions

During the rapid traffic growth in late 2013 and the first half of 2014, the ride‑hailing system faced severe LBS performance problems. Drivers reported GPS coordinates every few seconds, which were stored in MongoDB; passenger requests queried nearby drivers from MongoDB; orders were pushed to drivers via a long‑connection service; drivers accepted orders to start service.

Drivers upload GPS coordinates to MongoDB every few seconds.

When a passenger creates an order, nearby drivers are selected from MongoDB.

The order is pushed to the driver via a long‑connection service.

The driver accepts the order and begins the ride.

MongoDB, using a primary‑secondary replica set, suffered heavy read/write load (40k+ writes/s, 10k+ reads/s), leading to CPU spikes on secondary nodes, query latency over 800 ms, reduced throughput, and replication lag. The root cause was the 2.6.4 version’s global write lock and the large number of sub‑queries generated by LBS.

To mitigate this, the data was partitioned into four geographic regions, each with an independent MongoDB cluster, isolating traffic and reducing lock contention.

Long‑Connection Service Stability

The original long‑connection service used Socket for heartbeats and message push. Under peak load, the service became unstable.

Hardware issue: a single‑queue NIC caused one CPU core to run at 100 % while others remained idle, leading to packet loss and LVS connection drops. Replacing the NIC with a multi‑queue model resolved the problem.

Software issue: the Mina framework suffered from coarse‑grained memory control, difficult GC, inefficient idle‑connection checks, and CPU spikes under many connections. The service was rewritten with an AIO‑based framework, adding:

Custom development for the ride‑hailing scenario.

ByteBuffer pooling to reduce GC impact.

Buffer reuse across channels to avoid memory copies.

TimeWheel‑based idle‑connection detection to eliminate CPU spikes.

Priority‑aware data transmission.

Although Netty already provides pooling and TimeWheel, it lacks built‑in priority handling, which the custom AIO solution addressed.

System Distributed Refactoring

The original monolithic Web system combined HTTP services and TCP push services, leading to massive codebases, long build times, and low scalability. The system was split into three layers: business, service, and data.

Strong dependencies were handled with Dubbo for RPC and service governance; weak dependencies used RocketMQ for asynchronous messaging. Both Dubbo and RocketMQ are Alibaba open‑source projects, well‑understood by the team.

During the refactor, a unified development process, code standards, SQL conventions, and service degradation mechanisms were introduced.

Wireless Open Platform (KOP)

KOP was designed to solve client‑server communication problems:

Access control: each client receives an identifier and secret key for request signing; the server validates signatures before processing.

Traffic allocation and degradation: API rate limits can differ per client, with AB‑testing by city or platform; core APIs (login, order, dispatch, payment) are protected first.

Traffic analysis: real‑time detection of malicious IPs or users across multiple dimensions, with temporary or permanent bans.

Real‑time release: APIs can be enabled or disabled instantly without redeploying KOP, with an approval workflow.

Real‑time monitoring: per‑API metrics (total calls, successes, failures, avg latency) are collected per minute, visualized, and trigger alerts when thresholds are exceeded.

Real‑Time Computation and Monitoring

The team built a monitoring platform using Storm and HBase. The architecture (see image 2) consists of:

Core computation model: sum, average, group‑by.

Storm bolts: one parses logs into KV pairs; another applies rules to compute results, writing to RocketMQ every minute.

HBase storage: insert‑only design avoids row‑level locks; rowkeys are ordered for time‑dimensional queries, with periodic inserts keeping write rates low.

RocketMQ buffering: logs and Storm results are first placed in a MetaQ cluster; the buffer ensures stability under failures and traffic spikes, maintaining high TPS.

Sample real‑time business metric screenshots are shown in image 3.

Data Layer Refactoring

As business grew, a single database could no longer meet performance needs, especially for coupon and order services. The solution involved sharding, a generic framework for database partitioning, and a data‑sync platform.

Challenges addressed:

Data synchronization: front‑end databases (multiple) needed to map to a single back‑end database; MySQL replication alone was insufficient.

Offline batch extraction: frequent Sqoop dumps caused instability and duplicate extraction.

The sync platform (image 4) uses Canal to capture MySQL binlog changes (row mode), converts them to MQ messages, and distributes them via topics. Features include global and local ordering, replay at specific timestamps, monitoring, alerts, and automated node deployment.

Real‑Time Data Center

To handle massive data storage without changing existing applications, a real‑time data center was built on HBase (image 5). Front‑end MySQL shards are synchronized to HBase; a SQL‑to‑HBase parser converts queries, supporting secondary indexes via client‑side insertion.

Front‑end MySQL data is streamed to HBase.

A SQL parser translates SQL to HBase queries.

Secondary indexes are built by mapping fields to rowkeys; Coprocessor is avoided due to batch‑insert limitations.

Rowkey ordering defines query result order; reverse flag enables descending order.

Hash‑based rowkey dispersion spreads load across regions but disables range queries; hash takes precedence over reverse.

Composite ("串联") fields require unique indexes for multi‑table consolidation.

Additionally, an engine (HSQL) converts SQL statements directly to HBase API calls, offering low‑latency random reads, unlike Hive which targets batch, high‑latency workloads.