How JD Retail Tackles Data Governance Challenges to Boost Efficiency
JD Retail outlines the growing data management challenges it faces—including asset discovery, architecture agility, development quality, and rising IT costs—and presents a comprehensive data governance framework that leverages standards, agile architecture, development isolation, and resource optimization to improve efficiency and reduce operational expenses.
Data Management Challenges
Asset awareness weak : difficulty locating assets among hundreds of thousands of data models, many temporary, invalid, or duplicate tables.
Architecture not agile : coupling of metrics and dimensions, extensive pre‑computation, large budgets for iterative work, and wide tables consuming storage and compute resources.
Development quality and safety issues : uncontrolled table schema changes, operational risks from parameter mismatches, and development tasks that consume online resources.
IT resource cost rising : continuous growth of table count and storage, low utilization, and high proportion of cold or redundant data.
Data Governance System Construction
1. Standard Governance
JD Retail defined a unified data language standard covering business system, domain, theme, process, entity, attributes, update frequency, and granularity. Using this standard, high‑quality, high‑value models are certified and cataloged, while low‑quality models are decommissioned to free resources. Standard elements are systematized to improve dimension and metric registration, enabling metadata collection for downstream modeling and intelligent monitoring.
2. Architecture Governance
Adopt logical virtual tables to model dimensions and metrics, making the architecture more agile. Intelligent materialization (HBO, CBO, RBO) automatically decides which tables to pre‑materialize, reducing manual effort and IT cost. JD Retail also explores lake‑warehouse integration with incremental state updates and stream‑batch convergence to improve processing efficiency.
3. Development Governance
Implement development‑production isolation by separating accounts, tables, and queues, ensuring safe data production.
4. Resource Governance
Manage storage through table lifecycle, invalid/duplicate table identification, data compression, and migration. Manage compute by identifying and shutting down invalid tasks, optimizing low‑utilization jobs, handling frequent failures, and optimizing operators and engines, including peak‑shaving execution.
By mining proactive metadata and building governance models, JD Retail visualizes governance, enabling data governance to be systematic, efficient, and sustainable.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
DataFunTalk
Dedicated to sharing and discussing big data and AI technology applications, aiming to empower a million data scientists. Regularly hosts live tech talks and curates articles on big data, recommendation/search algorithms, advertising algorithms, NLP, intelligent risk control, autonomous driving, and machine learning/deep learning.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.