How Didi Scales HBase for Real‑Time Orders, Geo‑Tracking, and ETA

Background

HBase, built on the Hadoop ecosystem, serves both offline batch jobs and online latency‑sensitive services such as order and driver‑passenger trajectory queries. Offline tasks handle large‑scale data warehousing, while online services require fast, random access.

Multi‑language Support

Didi provides various language interfaces for HBase, including Java native API, Thrift (C++, PHP, Python), Phoenix JDBC, Phoenix QueryServer, MapReduce, Spark, and Streaming.

Data Types Stored in HBase

Statistical and reporting data: small volume, flexible SQL queries via Phoenix.

Raw factual data: orders, GPS traces, logs; large volume, high consistency, low latency.

Intermediate results: model training data; large volume, high throughput for batch queries.

Backup data for online systems: disaster‑recovery copy of relational or file‑service data.

Use‑Case Scenarios

Scenario 1: Order Events

Recent orders are cached in Redis; older or Redis‑unavailable queries fall back to HBase. Requirements include online status queries, historical order details, offline analysis, and throughput of 10 K writes/s and 1 K reads/s with 5 s data availability.

Rowkey designs:

Order status table: reverse(order_id) + (MAX_LONG - TS) Order history table:

reverse(passenger_id|driver_id) + (MAX_LONG - TS)

Scenario 2: Driver‑Passenger Trajectory

Supports real‑time or near‑real‑time coordinate queries, large‑scale offline analysis, and geographic range queries. GeoHash converts latitude/longitude to strings representing rectangular areas, enabling privacy‑preserving indexing and caching.

Range queries may intersect GeoHash blocks partially; points inside intersected blocks are filtered by exact distance to the query center.

Rowkey designs:

Single‑user query: reverse(user_id) + (Integer.MAX_LONG - TS/1000) Range trajectory query:

reverse(geohash) + ts/1000 + user_id

Scenario 3: ETA Service

ETA (estimated time of arrival) originally offline, now real‑time via HBase as a key‑value cache, reducing training time and enabling multi‑city parallelism.

Process:

Spark trains models every 30 minutes per city.

First stage reads all city data from HBase within 5 minutes.

Second stage completes ETA calculation within 25 minutes.

HBase data periodically persisted to HDFS for new model testing.

Rowkey: salting + city + type0 + type1 + type2 + TS Column families:

order, feature

Scenario 4: DCM Monitoring Tool

Monitors Hadoop cluster resources (NameNode, YARN containers). Uses Phoenix for SQL analytics on data collected by scheduled ingestion jobs, storing results in HBase for sub‑second query response.

Additional UI screenshots (figures 7‑9) illustrate usage statistics and are omitted for brevity.

Multi‑Tenant Management in Didi’s HBase

Didi adopts a single‑cluster multi‑tenant approach, using namespaces and RS Group to isolate resources, data, and permissions. The DHS (Didi HBase Service) system handles project lifecycle, resource estimation, permission management, and table‑level monitoring.

Key DHS modules:

Project lifecycle management (initiation, resource request, adjustments).

User and permission management.

Cluster resource management.

Table‑level usage monitoring (read/write, memstore, block cache, locality).

RS Group creates logical sub‑clusters by assigning specific RegionServer lists to groups, enabling exclusive or shared resource pools and reducing inter‑group interference.

Conclusion

Successful HBase adoption at Didi hinges on proper table schema design and disciplined resource control. Guiding users to understand HBase internals and providing clear resource isolation via DHS and RS Group reduces operational issues, lowers maintenance costs, and enhances overall platform stability.