JD.com’s Big Data Governance: Practices, Key Technologies, and Future Outlook

Introduction This article shares JD.com’s exploration and practice in big‑data governance, outlining the background, challenges, solutions, key technologies, resource‑optimization strategies, and future directions.

01 Background and Plan

In today’s data‑driven era, data has become a critical production factor with increasing strategic value for businesses like JD.com. The company operates tens of thousands of servers, exabyte‑scale storage, millions of data models, and millions of tasks, incurring double‑digit‑million‑level annual costs. To address rising cost pressure, JD.com seeks a scalable, continuous governance system rather than ad‑hoc firefighting.

The challenges include complex scenarios, evolving control rules, legacy jobs that bypass platform tools, diverse user cost awareness, and high manual governance costs and risks.

To tackle these, JD.com designed a health‑score and monetized billing to quantify governance benefits, and built an automated governance platform that discovers issues, notifies users, and executes fixes with one‑click actions while measuring impact.

Cross‑validation of multiple data sources (HDFS, Hive audit logs, metadata, lineage) to avoid mis‑judgments.

Multi‑stage verification to aggregate diagnostics over consecutive days, reducing false positives.

Real‑time validation of job submissions, with secondary checks for delayed offline models.

Reversible operations via automatic backups and one‑click rollback.

Governance mechanisms such as dedicated data‑management teams and clear role responsibilities.

Clear targets: annual governance goals are broken down to business units, departments, quarters, and months, with regular review meetings.

Incentive and penalty mechanisms to encourage compliance.

The system now covers cost, stability, security, and quality governance items, e.g., table lifecycle recommendations based on actual access patterns, dependency‑missing checks, security‑label accuracy, and metadata completeness.

02 Key Technologies

1. Audit Logs Audit logs record who accessed which data, when, and how, forming the foundation for security governance. JD.com extended native APIs to attach task IDs and sources, and performed content reverse‑engineering to distinguish read/write operations and associate actions with responsible owners.

2. Full‑Link Lineage By integrating Kafka‑based JDQ, Flink‑based JRC, DTS, and import/export pipelines, JD.com builds end‑to‑end lineage across production, data‑warehouse, and service layers. This enables impact analysis, chain optimization, and operator‑level tracing.

3. Operator‑Level (Operator‑Level) Lineage Operator‑level lineage captures field‑level relationships, distinguishing direct references, transformations, and usage as join conditions. It supports fine‑grained governance such as duplicate storage detection and field‑level metadata management.

Standard fields abstract business concepts, linking them to physical columns; combined with operator‑level lineage, they allow automatic discovery of downstream dependencies without manual mapping.

04 From “Throttling” to “Opening”

Beyond cost‑saving, JD.com pursues resource opening: mixing workloads, shifting task execution times, and cross‑data‑center task migration to improve overall utilization without additional cost.

Workload mixing: offline jobs borrow idle online resources during off‑peak hours and vice‑versa.

Task shifting: schedule low‑priority jobs to nighttime slots, predict queue load, and dynamically adjust execution windows.

Cross‑data‑center migration: balance compute and storage loads across sites, considering network bandwidth and storage demands.

When these practices are fully realized, resource utilization stabilizes near the optimal line, dramatically reducing procurement needs.

03 Future Outlook

Real‑time detection and remediation, moving from offline diagnostics to proactive interception before business impact.

Intelligent governance, evolving from rule‑based to AI‑driven precise problem identification.

Full automation, aiming for a managed, hands‑off governance model.

These directions will continue to advance JD.com’s data governance capabilities.

Conclusion The presentation concludes with thanks to the audience.