Didi’s Three Sessions at COC Asia 2026: Object Storage, Lakehouse, AI‑Friendly

Apache COC Asia 2026 was held in Beijing as an official global series of the Apache Software Foundation, bringing together open‑source leaders to discuss practice and trends. Didi’s technical team delivered three focused sessions on large‑scale object storage, open lakehouse, and AI‑friendly open‑source projects.

Session 1: From HDFS Metadata Bottleneck to Hundred‑PB Object Storage – Architecture Evolution, Performance Optimization, and EC Deployment

Didi’s Ozone team described how they tackled HDFS metadata memory pressure, block reporting, and RPC queue bottlenecks in small‑file scenarios. They built a multi‑cluster, thousand‑node Ozone storage system with capacities of hundreds of petabytes and billions of files. The design uses a layered Ozone Manager and Storage Container Manager, RocksDB for metadata, and containerized data organization to improve flexibility for object storage, small files, and multi‑tenant use cases.

Key practices include multi‑cluster governance, metadata read optimization, and first‑frame latency improvement. By routing buckets to different Ozone clusters via a ViewFS‑based client, they keep per‑cluster file counts below one hundred billion. Introducing Follower Read and dynamically selecting the optimal Ozone Manager based on latency and appliedIndex reduced GetMetaLatency P90 by roughly five times. To address first‑frame latency jitter on HDD, they employed NVMe cache for the first stripe of EC Block Groups and mitigated I/O interference through lock concurrency, background scanning, data deletion, and peer‑to‑peer migration.

On storage cost, Didi migrated from triple replication to EC‑6‑3, cutting the replication factor from 3.0 to about 1.5 and freeing hundreds of PB. They also resolved production issues such as EC shard deletion backlog, safe‑mode judgment, pipeline cache, repeated refactoring, and large‑scale data migration, feeding the optimizations back to the Apache Ozone community.

Session 2: Evolution to Lakehouse – Cost‑Effective Hive Migration and Real‑Time Ingestion at Scale

Yu Hangxiang, head of Didi’s real‑time computing engine team and Apache Flink Committer, presented the migration from Hive data warehouses to an open lakehouse built on Iceberg. The solution supports both offline and real‑time pipelines, offering a unified, open data consumption layer across multiple storage back‑ends (HDFS, object storage) and compute engines.

Iceberg serves as the core table format, with a Catalog layer providing discovery and governance. Amoro handles table maintenance independent of front‑end writes, asynchronously performing file compression, snapshot expiration, and orphan file cleanup while scaling horizontally.

For real‑time ingestion, Kafka, DDMQ, and PublicLog streams feed Flink directly to Iceberg snapshots, enabling downstream engines to consume data without redundant storage or compute, improving ODS freshness and pipeline stability. The Binlog Server aggregates MySQL binlogs from multiple clusters, offering unified health checks and DOS replay, reducing resource and operational costs. GTID‑Set based integrity checks precisely locate missing transactions. Additional capabilities added include minor/major compaction, metadata pushdown, and asynchronous small‑file merging.

The migration workflow is formalized as a standard closed loop: scan, verify, dual‑run, replay, data compare, validation, switch, observe, and rollback. Full‑link change, benefit calculation, anomaly analysis, and audit have been productized and automated. End‑to‑end data freshness improved from hour‑level to minute‑level, PB‑scale storage was saved, compute costs dropped noticeably, and data readiness became more stable and predictable.

Future work will deepen Iceberg table‑format evolution, continue delivering low‑cost, high‑stability real‑time lakehouse, enhance cross‑storage and cross‑engine unified Catalog and table operations, and support more Data + AI scenarios through multi‑engine collaboration.

Session 3: AI‑Friendly Evolution of the Open‑Source Project Seata

Zhang Feng, senior software R&D engineer at Didi and Apache Seata PMC, discussed how Seata is evolving to be AI‑friendly. With agents now participating in architecture understanding, issue handling, code modification, testing, and PR submission, AI coding brings efficiency gains but also introduces challenges such as context loss, compatibility risks, and higher verification costs.

AI‑friendliness does not mean accepting more AI‑generated code; it means making the project reliably understandable, safely operable, and still governed by the open‑source community. Seata’s evolution is framed in four layers: human‑readable, machine‑searchable, agent‑operable, and community‑governed. Using the Saga state language as an example, the talk showed how natural‑language business intents are transformed into structured, verifiable, visualizable, and executable state‑machine definitions, turning model output into deliverable artifacts.

The Seata Knowledge Agent aggregates documentation, code, API specifications, and community discussions, preserving source, version, and temporal semantics. The compiled knowledge is then used by intelligent agents to generate concise context packages containing key decisions, constraints, unresolved issues, and source citations. Evidence tracing, problem clarification, diff review, and Git history checks ensure knowledge trustworthiness.

Looking ahead, the Saga Copilot will link business descriptions, knowledge retrieval, state‑machine generation, semantic validation, workflow simulation, and change review, forming a complete loop from project understanding to solution generation and contribution verification. The ultimate goal is to enable both human developers and intelligent agents to participate in open‑source collaboration safely and efficiently.

From hundred‑PB object storage to an open lakehouse system and AI‑friendly open‑source evolution, the three sessions covered storage, data computing, and AI collaboration, demonstrating continuous architectural advancement, open‑source innovation, and expanding community cooperation.