In this interview, Didi data governance lead Liu Chao discusses his career journey, the unique technical architecture of Didi’s big‑data governance system, cost‑driven pricing models, metadata management, lineage extraction, automation practices, and offers practical advice for enterprises seeking effective data governance.
Didi’s Elasticsearch platform, built on ES 7.6 and deployed on physical machines with containerized gateway and control layers, provides a multi‑tenant, high‑performance search service—featuring a user console, operational controls, ZGC‑based latency reductions, cost‑saving compression, custom security, real‑time cross‑datacenter replication, and a roadmap toward ES 8.13.
This article outlines Didi’s best‑practice approach to constructing a global ride‑hailing data metric system, covering business scenarios, metric categories, pain points such as definition and technical challenges, and a comprehensive solution involving organizational structure, processes, model design, tooling, timezone handling, and governance.
This article details Didi's evolution from a multi‑engine OLAP stack to a unified StarRocks solution, explains the design of global dictionaries and materialized views for real‑time dashboard acceleration, and shares performance results, challenges, and future optimization directions.
This article details Didi's evolution of OLAP systems, the adoption of StarRocks for high‑performance MPP analytics, and how materialized views, global dictionary mapping, and transparent acceleration were engineered to boost real‑time dashboard queries while outlining performance gains, challenges, and future optimization plans.
The article reviews Didi's late‑November 2023 service disruption, detailing the timeline of failures, official apologies, and expert analyses of six possible technical causes—including software bugs, server issues, third‑party failures, DDoS, other attacks, and ransomware—while highlighting the role of a Kubernetes upgrade and cost‑cutting pressures.
The article explains the common P0‑to‑PX incident severity hierarchy used in software development, detailing what constitutes a P0 crash versus lower‑level issues, notes that definitions can vary across organizations, and adds a personal perspective on Didi’s service reliability.
To achieve disaster-recovery and cross-data-center resilience, Didi progressed from a simple proxy double-write scheme to a sophisticated MQ-free multi-active Redis design that uses a dedicated syncer, shard-based loop prevention, op-id replay protection, conflict detection, and incremental AOF durability, ensuring low latency, no data loss, and consistent availability.
The article explains observability as the ability to infer any system state from external data, contrasts it with traditional monitoring, outlines challenges of high‑dimensional, high‑cardinality data and storage costs, and describes Didi’s hybrid MTL architecture that separates low‑ and high‑cardinality logs and metrics while linking them via TraceIDs to provide detailed, cost‑effective insight and streamlined debugging.
At Didi, observability and stability engineering combine automated, AI‑driven alarm generation, distributed tracing, and ChatOps‑based fault handling to manage micro‑service complexity, massive traffic spikes, and cross‑region operations, emphasizing systematic investment, AIOps evolution, and a recruitment call for backend and test engineers.
The article details how Didi’s ride‑hailing team built a full‑link load‑testing simulation‑degree measurement system that quantifies test coverage across five dimensions—interface, scenario, category, link, and module—using normalized metrics, traffic prediction, and scoring formulas to identify gaps, improve stability, and guide future capacity‑planning enhancements.
Didi hardened its massive Elasticsearch deployment—spanning 66 clusters and thousands of nodes—by adding a custom security plugin that authenticates requests at the cluster level, implementing a one‑click toggle and staged rolling upgrades, ultimately enabling authentication across all clusters and dramatically reducing data‑leak risk.
Didi’s ride‑hailing backend evolved from a monolithic service to a DDD‑split architecture and the DuKang framework, then standardized components, rendering gateways, function access, and logging to solve multi‑terminal inconsistency, complex rendering logic, cross‑process integration, and data visibility, creating a unified cross‑end rendering platform supporting over twenty pages.
At a June 2023 forum in Beijing, Didi partnered with leading Chinese universities to showcase award‑winning projects that applied deep‑learning positioning, long‑term pricing models, high‑precision two‑wheel navigation, in‑vehicle voice separation, and data‑driven autonomous‑driving, illustrating how academia‑industry collaboration turns cutting‑edge research into commercial value.
Didi balances ride‑hailing supply and demand by forecasting regional needs with time‑series and deep‑learning models, then optimally repositioning drivers through integer programming and refining policies via imitation and offline reinforcement learning, ultimately enhancing passenger experience and platform efficiency.
The article introduces Didi’s large‑scale ride‑hailing marketplace, explaining how mechanism design, decision intelligence, operations research, reinforcement learning, and causal inference are combined to create fair, efficient, data‑driven matching, routing, and incentive systems that tackle massive two‑sided market challenges.
This article traces the evolution of big data technologies from early concepts and Google research papers through the rise of Hadoop, examines the platform transformations of Alibaba, JD.com, and Didi, and offers practical stack‑selection guidance for medium‑ and small‑scale enterprises.
This article chronicles Cheng Wei’s path from a modest upbringing and early setbacks through his tenure at Alibaba to founding Didi, detailing the company’s early struggles, rapid growth, fierce competition with Uber, strategic mergers, and its eventual dominance in China’s ride‑hailing industry.
The article follows three Didi Ride‑Hailing engineers who joined in 2016 as fresh PhDs, detailing how they leveraged machine‑learning, dynamic dispatch, and product‑line automation to rise from junior developers to technical leaders, highlighting the blend of hard coding, soft communication, rapid‑learning culture, and the team’s current senior‑role hiring drive.
To overcome scarce labeled data for safety and governance, Didi combines few‑shot learning with systematic data augmentation, self‑training semi‑supervised labeling, and multi‑task neural architectures, cutting labeling costs and reducing log‑loss by over 20% while boosting ROC‑AUC and PR‑AUC across harassment detection, expense‑complaint, and route‑intercept use cases.
Didi migrated HDFS to Hadoop 3.2 and implemented erasure coding—using XOR and Reed‑Solomon RS(6,3) striping—to replace three‑replica storage for cold data, building back‑ported clients, automated conversion tools, and cross‑datacenter backup pipelines, while addressing operational bugs and noting performance trade‑offs.
The Youzan Big Data Technology Salon brought together Youzan, NetEase and Didi to share practical approaches for cutting data‑infrastructure costs, building an Apache Iceberg‑based data lake, scaling Flink real‑time workloads, and creating a data‑driven growth platform that leverages tracking, A/B testing and analytics.
This article presents a comprehensive catalog of more than 40 open‑source projects released by Didi, covering AI runtimes, speech and NLP engines, big‑data loaders, middleware, mobile frameworks, frontend UI libraries and various operational tools, each with a brief description and a GitHub link.
This article details Didi’s end‑to‑end construction of a real‑time data warehouse for the Ride‑Sharing (顺风车) business, covering motivations, layer‑by‑layer architecture, naming conventions, StreamSQL capabilities, operational tooling, achieved results, challenges, and future batch‑stream integration plans.
This article explains Didi's comprehensive methodology for constructing, operating, and productizing a metric system, covering metric definitions, lifecycle stages, OSM and AARRR modeling, scenario‑based design, management challenges, metadata governance, and tool support for data‑driven decision making.
This article details Didi's end‑to‑end real‑time data warehouse design for the carpool business, covering its objectives, architecture layers from ODS to application, naming conventions, StreamSQL development, operational tooling, challenges faced, and future batch‑stream integration plans.
Didi’s engineering team analyzed a severe write bottleneck in their 3000‑node Elasticsearch cluster, identified long‑tail latency caused by refresh, translog locks, write queues and GC, and applied routing‑aware bulk writes, JVM and Lucene tweaks, and data cleaning to more than double write throughput while slashing server costs.
The article introduces Didi’s IFX platform, describing its background, four‑layer architecture (access, software, engine, compute), productization efforts such as high‑performance optimizations, model and engine compression, unified deployment across hardware, multi‑framework support, automation, and security enhancements, and concludes with future plans.
Didi's HBase team upgraded eleven clusters from version 0.98 to 1.4.8, tackling maintenance burdens and custom‑patch divergence, validating RPC and HFile compatibility, performing extensive functional and performance tests, opting for a rolling upgrade, fixing a region‑split data‑loss bug, merging critical upstream patches, and establishing a reusable migration methodology.
This article explores Didi’s open‑source Chameleon framework, detailing its origins, cross‑platform goals, MVVM‑based design principles, architecture layers, DSL compilation, runtime implementation, engineering tooling, ecosystem components, and real‑world adoption across dozens of Didi products and external teams.
This article introduces Didi's adaptive grouping algorithm, which enhances the uniformity of user allocation in AB experiments by replacing traditional complete randomization with a single-pass method that balances observed metrics across groups, and demonstrates its effectiveness through large‑scale experimental results.
Didi has unified its real‑time computing on Apache Flink, creating an enhanced StreamSQL service with extended DDL, built‑in parsers and UDX, supporting thousands of nodes, millions of jobs, and trillions of daily records, while addressing state management, high availability, multi‑language UDFs, and pursuing real‑time ML and data‑warehouse integration.
Didi's National Carpool Day on Dec 3 2019 attracted 3.1M passengers; stability ensured via six pillars: organized task force, capacity forecasting and rapid container scaling, comprehensive monitoring with fire‑fighting map, robust contingency platform, strict process standards, and coordinated third‑party preparation.
This article presents a comprehensive overview of modern speech recognition technology, covering basic ASR concepts, classic acoustic and language models, deep‑learning approaches such as DNN‑HMM, CTC, attention‑based and transformer models, multimodal fusion, signal‑processing pipelines, and practical deployment considerations at Didi.
Didi’s Xiaoju Car Service leverages a reinforcement‑learning framework with Double DQN and graph‑embedding‑based personalization across its traffic‑distribution, tagging, portrait, targeting, strategy, and reach‑optimization modules, replacing manual rule‑based marketing, and achieves roughly 8 % new‑user lift, 50 % cost reduction, and significant gains in open and conversion rates.
This article describes how Didi Ride‑Sharing leverages reinforcement learning and graph‑embedding techniques to model and optimize user‑operation marketing, detailing system architecture, algorithm design, experimental ROI improvements, and personalized message delivery for enhanced conversion and cost efficiency.
This article examines the big‑data architectures of three leading Chinese internet companies—Taobao, Meituan, and Didi—detailing their data sources, synchronization mechanisms, batch and streaming processing layers, and the common scheduling components that unify their Hadoop‑based ecosystems.
The article presents a detailed overview of Didi's massive traffic data processing, its big‑data technology stack—including Hadoop, Hive, Flink, Spark, HBase, Presto and ClickHouse—intelligent data catalog, lean data production practices, and impressive operational metrics that illustrate how the company achieves high‑performance, real‑time analytics at scale.
The article examines Didi’s “Guess Where You’re Going” feature, describing its product benefits, the contextual data used, and a simple Bayesian Gaussian model that predicts a user’s destination based on time, departure location, and weekday, while also discussing its limitations and potential improvements.
This article analyzes how Didi’s unique ride‑hailing scenario big data is applied to automotive finance, detailing the business model, asset‑side and full‑process risk challenges, data‑driven solutions, and future prospects for intelligent credit risk control in both enterprise and retail lending.
The talk presents Didi AI Labs' research on intelligent dialogue robots for ride‑hailing, covering single‑turn QA, multi‑turn conversation, multi‑task learning architectures, model experiments, active learning pipelines, and the overall system design that integrates intent detection, slot extraction, dialogue management, and response generation.
Recent reports reveal that many Android apps secretly collect location and identifier data despite denied permissions, a zero‑day Zoom flaw lets any website hijack Mac cameras, and Didi adjusts its pricing, while a man released after splashing water on Baidu's CEO highlights broader tech‑related news.
The article outlines Didi’s rapid growth in open‑source contributions—20 projects, nearly 40 000 stars, 130+ contributors across AI, middleware, monitoring, mobile and front‑end—detailing its internal governance, external patches to major OSS projects, community partnerships, and future open‑source commitments.
The article provides an in‑depth overview of Didi’s Intelligent Mobility Open Platform, detailing its AI foundation, open‑source projects, a suite of AI services for voice, vision, mapping and safety, three smart‑mobility solution tracks, and the company’s strategic plans to expand the platform globally.
This article describes how Didi adopted Apache Flink for its real‑time data streams, detailing the platformized architecture, production use cases such as ETL, monitoring and CEP, the evolution of StreamSQL, and the engineering improvements made to support large‑scale, low‑latency processing.
The article explains the Weber‑Fechner law and its role in Prospect Theory, then shows how Didi applies these concepts—using a log‑linear order‑distance model and reference‑point‑based strategy evaluation—to reduce cancellations, improve driver perception, and guide data‑driven product decisions.
Chief Scientist Tang Jian explains how Didi leverages next‑generation AI—big‑data, hybrid‑augmented, autonomous, and collective intelligence—to transform smart mobility through advanced dispatch, safety systems, in‑car perception, traffic‑signal optimization, and global collaborations, while confronting challenges of model scale, computing power, and safety assurance.
This article details Didi's 2016 full‑link load‑testing initiative, covering data‑isolation strategies, virtual driver/passenger tooling, trace‑based traffic marking, staged deployment tactics, and the operational insights gained from stress‑testing a massive ride‑hailing platform.
The article explains the launch of the world’s first DevOps capability maturity model, the collaborative effort behind it, Didi’s role as a standards workgroup member, and how its OE (OneExperience) platform embodies the new guidelines to streamline the entire software delivery lifecycle.
Didi’s vice‑president and chief scientist of smart transportation, Professor Henry Liu, delivered the “Big Data‑Driven Smart Transportation” lecture—part of the AI Industry Applications course—on China University MOOC, teaching students fundamental concepts, real‑world cases, and future prospects of big‑data and AI in traffic management.
The article details Didi's end‑to‑end load‑testing strategy—including online environment testing, data isolation with virtual orders, trace‑based traffic marking, and a distributed virtual driver/passenger tool—describing its design, deployment stages, findings, and future reliability applications.
In this interview, Didi R Lab senior engineer Du Huan discusses his decade‑long Go experience, the language’s low‑cost entry, system‑level performance and cloud‑native strengths, upcoming enhancements like generics, his large‑scale microservice framework design for Gopher China 2019, and practical advice for developers and enterprises adopting Go.
In 2018 Didi expanded its open‑source portfolio to 12 projects, amassed over 16,000 stars and 2,000 forks, contributed more than 300 pull requests, engaged 65 contributors, and delivered dozens of patches to major projects such as Hadoop, Spark, Flink, and Linux, while deepening community partnerships and outlining ambitious open‑source goals for 2019.
Didi transformed its massive single‑cluster Elasticsearch deployment into a transparent multi‑cluster architecture using TribeNode and cross‑cluster search, isolating workloads, reducing fault impact, and achieving five‑fold scale while preserving a single‑cluster appearance for services, despite added configuration complexity and stability challenges.
Facing growing data‑driven pressures, Didi designed a full‑link data tiered protection framework that defines classification standards, integrates data levels across the entire pipeline, and applies concrete safeguards and tooling to improve resource allocation, backup reliability, and overall data reliability.
Using real‑time order and fuel‑point data from the Didi driver app, the author demonstrates a systematic, data‑driven approach to identify the most cost‑effective car models for ride‑hailing across major Chinese cities, complete with methodology, analysis, and city‑specific rankings.
This article outlines Didi's MySQL database architecture, the challenges of managing thousands of instances, and the step‑by‑step automation framework—including dbproxy, high‑availability, backup, monitoring, and deployment modules—that reduces manual DBA work by over 70%.
At GopherChina 2018, Didi engineer Shi Songran explained how Go powers the company’s high‑availability order, dispatch, and storage systems, detailing service‑governance practices, two production‑level bugs, and Didi’s open‑source contributions of the gendry database helper and the high‑performance json‑iterator codec.
This article recounts Didi’s Elastic Cloud journey, explaining why a private cloud was needed, why containers and Kubernetes were chosen, detailing the overall architecture, product features, networking, monitoring, logging, storage solutions, and sharing practical insights and future plans.
This article explains how Didi leverages HBase’s distributed architecture, multi‑language APIs, and custom rowkey designs to support online order queries, driver‑passenger trajectory tracking with GeoHash, real‑time ETA calculations, and a monitoring platform, while managing multi‑tenant resources through DHS and RS Group.
Didi’s Apollo platform combines A/B testing with gradual (gray) release, letting product teams safely roll out new features to targeted user segments, monitor key metrics, and apply best‑practice guidelines—such as isolating variables, pre‑defining metrics, controlling duration, random grouping, and confidence analysis—to achieve statistically significant, data‑driven improvements across thousands of weekly releases.
