This article provides a comprehensive guide to designing the Data Warehouse Detail (DWD) layer, covering Kimball‑based design principles, step‑by‑step modeling, table and field naming conventions, concrete Hive DDL/DML examples, and optimization techniques such as partitioning, bucketing, and compression.
This article provides a comprehensive guide to designing and implementing a data middle platform, covering its core essence as a business-data fusion mechanism, six architectural modules from ingestion to performance optimization, three implementation phases from requirements to applications, and key pitfalls like business-technology misalignment and over-engineering.
This article defines data architecture per ISO/IEC/IEEE 42010 and DCMM, outlines core components — data model, flow, metadata, integration — identifies four key challenges including siloed management and technology-first approaches, and proposes optimization paths: enterprise-wide architectural governance, layered stable design, and collaborative data governance.
This case study details how the BIP procurement system tackled massive data volume, complex queries, and slow SQL by segmenting inbound orders, leveraging Elasticsearch, introducing a dynamic routing layer, and implementing robust ES high‑availability and monitoring, ultimately reducing query load by over 90%.
Data ModelingPerformance OptimizationQuery Optimization
The article examines current limitations of AI agents in planning and complex reasoning, critiques existing methods like COT/TOT and ReAct, and proposes practical strategies—including combined COT‑Reflection approaches, structured memory algorithms, and white‑box interaction designs—to enhance agent performance within the DataFun knowledge map framework.
The article explains the three detailed-granularity fact table types—transaction, periodic snapshot, and cumulative snapshot—detailing their purposes, design principles, and comparative usage, and offers a simplified interpretation to help data engineers choose the appropriate fact table for data warehouse modeling.
This article offers a step‑by‑step guide for programmers to quickly onboard a new business or project by gathering documentation, mapping data models, understanding architecture, learning platform tools, and applying practical tips to accelerate learning and growth.
This article explains how major payment institutions ensure the accuracy of tens of millions of daily transactions and billions of dollars by using a four‑segment data model, three verification groups, error classification, and extensible data coding to achieve reliable settlement and accounting.
The article outlines Didi’s end‑to‑end metric lifecycle—from background, requirements and current pain points to a multi‑stage solution that introduces a unified metric dictionary, management tool, logical modeling, and consumption layer—to achieve accurate, timely, consistent, and efficiently managed indicators across the data warehouse ecosystem.
This article provides a comprehensive guide to using Scikit-Learn for data modeling, covering linear regression and decision tree algorithms, including data preparation, model training, evaluation metrics, and visualization techniques for predictive analysis.
This article introduces feed streams, explains their evolution from RSS to modern social feeds, classifies them by aggregation logic and display, discusses challenges such as real‑time performance and massive data, and presents a backend architecture with data models, pagination, write/read diffusion, and core publishing/reading workflows.
Facing inconsistent metrics and low reuse in siloed data services, the team built a unified metric service using a headless BI semantic layer and virtual data models, enabling consistent metric definitions, reusable data models, AI-friendly queries, and faster, scalable reporting across the organization.
This article introduces core relational database concepts, covering tables, data models and ER diagrams, the fundamentals of SQL, and the first three normal forms, providing clear explanations and examples to build a solid foundation for further database study.
This article presents Kuaishou's metric middle platform, detailing its background, design principles, architecture, metric management, data modeling, unified analysis language OAX, federated query engine OCTO, acceleration strategies, and future directions, illustrating how it improves data quality, development efficiency, and analytical capabilities at scale.
This article explains how data modeling drives business value by sparking conversations about data meaning, enabling the creation of useful data objects, and guiding smart decisions on data capture, storage, usage, and integration, while also outlining a governance framework for managing enterprise data models.
This article compiles comprehensive MySQL best‑practice guidelines covering charset selection, storage engine choice, naming conventions, field design, index optimization, SQL development tips, and operational procedures to improve performance, maintainability, and reliability of database systems.
This tutorial shows how to model nationwide train schedules as a graph, prepare CSV data, import it into Huawei Cloud GES, and write Cypher queries that discover faster, more flexible transfer routes than the official 12306 service, complete with Python code examples and visual results.
This article outlines Didi's approach to metric management, covering background, data product overview, and challenges in traditional and agile BI models, and presents a comprehensive solution for metric standardization, logical modeling, quality assurance, unified consumption, and future roadmap to improve data warehouse efficiency and consistency.
This guide outlines a practical six‑step methodology—covering overall planning, data integration, model construction, data development, asset management, and data services—to help enterprises build a robust data middle platform that unlocks business value and supports agile digital transformation.
This article walks through the fundamentals of data modeling, explaining what data is, the DIKW framework, why modeling matters, and detailing the end‑to‑end process from conceptual design through logical and physical layers, including DIM, DWD, DWS, and ADM tables with practical tips and naming conventions.
This article recounts L's attempt to change a train ticket, analyzes seat availability across four segments, and presents a minimal SQL model with table creation, data insertion, and a query that determines feasible seat combinations, demonstrating how segment‑wise seat allocation can enable complete journey booking.
Drawing from the book “Huawei Data Way”, this article explains why Huawei must digitize, describes the resulting data‑island problem, and outlines the four‑part framework of data governance—including data asset catalogs, standards, models, and distribution—while showing how business‑object‑centric information architecture is built and implemented.
This article presents a comprehensive overview of Ant Group's metric system, covering its definition, three-layer architecture, common challenges, concept consensus methods, semantic layer options, mechanism design, productization capabilities, platform improvements, business outcomes, future directions, and a detailed Q&A session.
This article reviews two decades of database research, analyzing the rise and decline of various data models—from hierarchical and relational to NoSQL, vector, and graph databases—while highlighting how AI, cloud, and hardware advances are reshaping DBMS architecture and predicting which approaches will dominate tomorrow’s data landscape.
This article presents Kuaishou's metric middle platform, detailing its background, design principles, metric management and service architecture, including headless BI concepts, unified analysis language OAX, query engine OCTO, data modeling layers, acceleration strategies, and future directions toward intelligence and high performance.
Data architecture governance, encompassing standards, security, modeling, quality, and lifecycle management, is essential for digital transformation in fast‑growing industries like express delivery, and this article outlines current challenges, traditional approaches, and a practical, phased methodology with platform support to implement effective governance.
This article presents Ant Group's data‑driven approach to digital transformation and risk control in the bulk commodity sector, covering background challenges, data‑application pain points, core capabilities, credit‑risk models, data‑asset construction, indicator frameworks, and secure data integration for B2B scenarios.
This article explains how companies can integrate independent user‑profile tag systems into a unified framework, covering tag definitions, demand sources, classification, construction methods, update cycles, platform architecture, common algorithms, and practical applications such as marketing, KPI attribution, and A/B test analysis.
This article outlines the step‑by‑step evolution of Baidu's Feed data warehouse—from traditional layered modeling to hour‑level core tables, then real‑time wide tables, and finally a flow‑batch integrated multi‑version wide‑table architecture—highlighting the motivations, design choices, challenges, and resulting benefits.
Baidu’s Mobile Ecology team transformed its Feed data warehouse through three progressive stages—hour‑level core tables, a real‑time wide table, and a unified day‑level multi‑version table—consolidating traffic, content, and user data into a single partitioned wide‑table architecture that resolves granularity inconsistencies, cuts processing cost, and delivers real‑time to daily latency for diverse analytics.
This article explains the concept of data standards, outlines the three main categories of data standards, describes a four‑stage implementation process, and provides a real‑world bank case study to illustrate how enterprises can establish and apply data standards for better data quality and value.
This presentation by senior data warehouse engineer Huang Luo details Wing Payment’s end‑to‑end data warehouse build, covering background challenges, governance framework, platform architecture, layered modeling, naming standards, asset management, monitoring, and future plans, illustrating how systematic data governance drives cost reduction, efficiency, and security.
This article details JD Retail’s end‑to‑end data platform, covering data asset certification, 5W2H modeling, unified query DSL, intelligent acceleration, robust governance, visualization components, low‑code orchestration, and large‑model AI applications that together reduce query latency, cut development costs, and empower analysts across the retail business.
This article examines why data departments get overwhelmed by massive data‑retrieval requests, identifies root causes such as mindset, requirement handling, and lack of tools, and presents a technical solution centered on dimensional modeling and OLAP multi‑dimensional reporting to streamline data access and empower teams.
This article details how the Dongchedi business team designs, implements, and monitors a comprehensive metric system within its data warehouse, covering metric standards, model construction, metadata management, quality monitoring, application scenarios, and future directions using the DataLeap platform.
This article describes how a C++‑based cross‑platform framework and a unified routing mechanism were built for WeChat Pay to eliminate platform‑specific bugs, streamline data flow, improve crash stability, cut code size by nearly 45%, and boost development productivity across iOS and Android.
This article explores the complete lifecycle of an indicator system—from defining metrics and addressing common ambiguities, through designing concept consensus, semantic layers, mechanisms, and governance, to productizing platforms, optimizing development, and envisioning future AI‑driven enhancements.
This comprehensive guide explains the purpose of architecture diagrams, the criteria for good diagrams, the optimal moments to create them, and detailed methods for drawing business, application, technical, code, and data architecture diagrams, complete with design principles, classification, and practical tips.
Data ModelingTechnical Architecturearchitecture diagrams
This article explains how to build and apply a comprehensive metric system by addressing business, technical, and product challenges, outlining design, standardization, metadata management, and future AI‑driven use cases to support data‑driven decision making.
Grey Theory, introduced by Deng Julong in 1982, offers a mathematical framework for analyzing systems with incomplete or uncertain data, using techniques like generated series and the GM(1,1) model to enable reliable forecasting and decision‑making across fields such as economics, environment, and product lifecycle analysis.
The article explains retention analysis models, their importance for user loyalty, outlines offline Hive architecture, then shows how ClickHouse’s retention() function and columnar storage dramatically speed up multi‑day retention calculations, providing SQL examples and practical guidance for product analytics.
This article explains the concept of a metrics layer, why StarRocks is suited for building such platforms, and presents detailed case studies from Airbnb, a major bank, and a leading restaurant chain, while comparing three modeling paradigms and outlining the future vision for materialized views.
The article explains how to analyze, decompose, and design a payment system by breaking the workflow into modules, illustrating pre‑payment, third‑party integration, and post‑payment stages with sequence diagrams, and proposing a data‑structure model that covers account management, transaction records, order handling, and related business components.
This article outlines the securities sector's data‑governance background, vision, common pain points, and a comprehensive service‑oriented framework covering data models, quality, standards, metadata, and security, and demonstrates the resulting business value and future outlook.
This article explains how to design and implement a traffic distribution diagram for an observability system, covering current cloud‑native tooling, data standardization, transformation, traffic‑flow modeling, aggregation, storage with ClickHouse, and visualisation techniques such as Sankey diagrams.
This article explains how Heartbeat Game created a cloud‑native big data platform on Alibaba Cloud using DataWorks, detailing the end‑to‑end data pipeline, a universal logical data model for games, and the advantages of the GS‑LBDM architecture for AI, risk control, and analytics scenarios.
This talk by Ant Group’s senior technical expert Wang Gaohang details the definition, design, mechanism, productization, and future outlook of the company’s metric system, covering concept consensus, semantic layers, workflow, AI assistance, performance optimization, and practical case studies.
This article explains how agile methodologies can be applied to data management, covering the need for agile data practices, core principles, iterative modeling, governance, CI/CD pipelines, tooling, metrics, security, case studies, challenges, and future outlooks in a comprehensive, step‑by‑step guide.
Discord’s rapid growth forced a series of massive database migrations—from MongoDB to Cassandra in 2017, then to ScyllaDB in 2023—detailing the motivations, requirements, data‑model design, performance challenges, migration tooling, and the resulting operational improvements.
This article presents a comprehensive overview of Xiaomi's sales data warehouse, covering its evolution, dimensional modeling and layer theory, Lambda architecture with batch and streaming processing, capability layers, security measures, and future trends toward real‑time metricization and data value creation.
This article details the background, challenges, design principles, and implementation of a unified data service platform—including data modeling, multi-source governance, real-time processing, and a Doris-based storage solution—to support large‑scale video data for a new media operation.
This article explains how to redesign a Java data model so that a single sessionId is stored at the outer level while multiple sceneId/sceneName pairs are nested inside a sceneList, and shows the required MyBatis mapper, XML mapping, service, and controller implementations.
This article explains what NoSQL databases are, outlines their typical use cases and architectural components, compares them with traditional relational databases across storage, scalability, query, and transaction aspects, and highlights the advantages and trade‑offs to consider when choosing a data solution.
This article details NetEase EasyData's evolution and product matrix, explains why a metric middle platform is needed, describes its core technical architecture—including a unified logical semantic model, a custom metric query language, and engine decoupling—and outlines future development directions.
The article details OPPO’s approach to balancing cost reduction and user value in app distribution through unified cross‑scenario modeling, sparse‑data solutions, oCPX advertising optimization, and a multi‑stage smart‑power system that improves efficiency, scalability, and revenue while preserving user experience.
The article explains the differences between data schemas, data structures, and data models, proposes a systematic naming approach, and outlines a five‑schema architecture—including business, view, logical, deployment, and physical schemas—while addressing terminology challenges and normalization processes.
This article introduces NoSQL databases, explains their core concepts, typical use cases, architectural components, and contrasts them with relational databases, helping readers understand when and why to choose NoSQL solutions for large‑scale, unstructured data workloads.
The article details JD Logistics' order system redesign using a four‑layer transaction architecture, describing its decoupled backend, unified data model, high‑availability components such as CQRS, Redis, JMQ, HBase, and Elasticsearch, and outlines design advantages, extensible data modeling, future challenges, and overall performance outcomes.
Backend ArchitectureData ModelingDistributed Systems
This article introduces Alibaba Cloud DataWorks' intelligent data modeling tool, outlines the data demand flow, shares best practices and hands‑on demonstrations for data warehouse modeling, discusses common challenges and their solutions, and provides Q&A and product details for developers and data engineers.
This article explains what a data warehouse is, contrasts it with traditional databases, outlines the evolution from classic to internet‑scale warehouses, details modeling approaches and layered architectures, discusses KPI dictionaries, date dimensions, naming standards, data governance, incremental loading techniques, and upstream/downstream coordination.
This article explains how to optimize OLAP workloads by balancing normalization and denormalization, applying data sharding, replication, indexing, partitioning, materialized views, columnar storage, compression, and lifecycle management, and provides practical ClickHouse SQL examples for index creation, partitioning, and query plan analysis.
This article outlines how a marketing‑focused data middle platform can be constructed by integrating online and offline behavior data, business data, and third‑party sources, then applying data integration, modeling, processing, and application capabilities to enable data‑driven user journeys and personalized marketing strategies.
This article walks through a real‑world e‑commerce scenario where billions of daily browsing events generate over 100K TPS writes, evaluates storage options on reliability, cost, read/write performance and implementation difficulty, and ultimately recommends Tair after detailed analysis of List, Sorted‑Set and Hash structures, code examples, and concurrency controls.
This article details the design of a decoupled, high‑availability order system, covering business scope, value propositions, layered architecture, real‑time data layer, read/write separation, caching, messaging, search, multi‑tenant support, data security, and future challenges such as personalized queries and cost‑effective scaling.
The article explains that, although MySQL’s theoretical limits are far higher, practical B+‑tree page and index calculations suggest keeping a single table under about twenty million one‑kilobyte rows to avoid performance degradation, illustrated with interview examples and estimation formulas.
This guide explains how data engineers can rapidly adapt to a new workplace by mastering business context, data domains, and system architecture, using structured learning, practical case studies, and continuous reflection to earn trust and deliver value efficiently.
This article compares MySQL and Elasticsearch across data models, query languages, indexing, distributed architecture, performance, scalability, and typical use cases, highlighting their distinct strengths and trade‑offs to help developers decide which system—or combination—best fits specific application requirements.
This article compares SQL and NoSQL databases, outlining their advantages, limitations, and ideal use‑cases, and provides guidance on selecting the appropriate technology based on consistency, scalability, and data model requirements.
This article introduces graph databases as a NoSQL solution, explains property‑graph modeling, compares relational and document stores, evaluates several graph products, and details Dgraph’s architecture, indexing, query language, and real‑world business applications such as knowledge graphs and equity‑relationship analysis.
This article explains why data‑warehouse standards are essential for improving team efficiency, product quality, and maintenance costs, and provides a step‑by‑step guide covering standard creation, discussion, rollout, supervision, continuous improvement, as well as detailed design, process, quality, and security specifications.
The article presents Tencent Oula's comprehensive data‑asset platform that integrates data collection, integration, warehouse and metric modeling, unified services, and a governance engine to create trustworthy, low‑entropy data assets while addressing common data‑governance challenges and outlining future AI‑for‑BI possibilities.
This guide presents a comprehensive methodology for building knowledge graph schemas that decouple structural representation from semantic meaning, covering schema design, attribute semantic standardization, concept modeling, multi‑relational and hypergraph techniques, and practical steps for implementation across complex business domains.
This article presents a comprehensive methodology for building an Onedata data warehouse, covering the conceptual framework, data modeling processes, the Inmon and Kimball approaches, practical case studies from Baidu, Huawei, and banking, and key takeaways for enterprise data architecture.
Designing a time axis for HR systems involves choosing between an interval model that stores start and end timestamps and an effective‑date model that uses effective dates and sequences, each affecting query simplicity, maintenance effort, common pitfalls, and ultimately enabling accurate historical and future employee data analysis.
This guide walks through using the TuGraph graph database to design and deploy graph applications, covering real‑world scenarios, database selection, built‑in datasets, Docker/CentOS/Ubuntu deployment, model design, data import, debugging, operational monitoring, and integration with services or direct RESTful APIs.
The article traces the evolution from Turing's seminal AI test through the rise of ChatGPT, explains how large language models rely on knowledge graphs built from massive unstructured data, and examines the challenges and advantages of modern graph databases for high‑performance, flexible, and explainable AI applications.
The paper introduces SASS, a scenario-adaptive self-supervised recommendation model that uses contrastive pre-training and multi-layer gating to expand global samples and transfer scene-aware parameters, enabling a single model to deliver personalized recommendations across diverse Taobao ‘SuoSuo’ scenarios while mitigating data sparsity and cross-domain challenges.
This article explains data architecture and data modeling fundamentals, presents DAMA DMBOK concepts, outlines four practical solutions for model design, standard management, automated change control, and business mapping, and shares an enterprise manufacturing case study with Q&A on governance and efficiency.
This article presents a detailed roadmap for enterprise data governance, covering business digitization goals, data governance construction, typical digital platform architecture, core governance actions, implementation pathways, data asset inventory techniques, and real‑world case studies to illustrate practical execution.
This article outlines DeepEx Technology's comprehensive data governance methodology, covering construction paths, digital platform frameworks, core governance components, implementation steps, case studies, and a Q&A that together illustrate how enterprises can build reliable data assets, models, standards, and quality processes to unlock business value.
These COMAP MCM problem statements challenge teams to develop statistical models for pricing used sailboats using a large 2023 dataset and to propose innovative strategies for the Olympic Games, evaluating regional effects, data sources, and policy recommendations for sustainable hosting.
The article explains the purpose and benefits of data publication diagrams for visualizing relationships among data entities, services, and application components, introduces UML/BPMN EAP profile components with illustrative images, and outlines Archimate modeling elements, highlighting how data is localized within repositories or entity application components.
This article explains the concept of data models, why warehouse models need reconstruction, compares normative and dimensional modeling approaches, and provides a step‑by‑step guide—including information gathering, design, and implementation—to build efficient, maintainable data warehouse architectures.
This article explains what data models are, why model refactoring is necessary, compares normalized and dimensional data warehouse modeling approaches, and details a three‑step implementation process—including information research, model design, and model deployment—while highlighting best‑practice naming conventions and practical examples.
To handle exploding product inventory data, the company overhauled its warehouse inventory model by eliminating risky document‑hand‑offs, storing only changed rows instead of daily snapshots, and syncing transformed data to a data‑warehouse for reporting, which cut monthly accounting time by 30 hours (≈30 %), improved accuracy, enabled new analytics, and introduced TiDB migration and team upskilling.
This article details the challenges of fragmented ODS data in the moving‑service domain and explains how a dedicated public‑layer data warehouse, with layered architecture and quality monitoring, was designed and implemented to improve data reuse, reduce redundancy, and stabilize downstream analytics.
This article details Bilibili's 北极星 user behavior analysis platform, tracing its evolution from early Spark‑Jar models to Flink‑ClickHouse pipelines and Iceberg‑based full aggregation, and explains the technical solutions for event, retention, funnel, path analysis, data ingestion, cluster rebalancing, and performance optimizations that enable massive real‑time analytics on billions of daily events.
This comprehensive guide walks you through the entire data governance ecosystem, covering metadata fundamentals, classification, maturity models, data standards, modeling, integration, lifecycle management, quality assurance, security, and ETL processes, all illustrated with clear diagrams and practical steps.
This article outlines the design and implementation of a unified data service platform that standardizes data access, accelerates model processing, provides flexible API construction, and ensures high availability through gateway, caching, and monitoring, ultimately reducing cost and improving efficiency for both C‑end and B‑end applications.
The article outlines a standardized data‑service platform built atop a warehouse, detailing its construction, query, and gateway layers—supporting model definition, acceleration, reusable APIs, unified DSL/SQL interfaces, and observability—to solve ingestion, definition, and lineage issues, achieving 500+ APIs, sub‑day creation, and 18% cost reduction.
This article explains the concepts, architecture, and layered design of data warehouses, covering data flow, ETL processes, ODS, DWD, DWM, DWS, ADS layers, their characteristics, differences from databases, and the role of data marts in supporting OLAP and decision‑making.
This article explains the concept of Lifetime Value (LTV), presents a two‑step estimation method based on purchase pathways, evaluates model accuracy with real data, and discusses practical applications such as acquisition guidance, fine‑grained operations, and product iteration decisions.
This article details Xiaomi's multi‑year journey in building a group‑wide Business Intelligence platform, covering its historical evolution, technical challenges in performance, modeling, visualization and permissions, the current four‑layer architecture, and future plans to make the platform more business‑centric and simpler.
This comprehensive guide explains why data modeling is essential for big‑data systems, compares relational and OLAP approaches, details ER, dimensional, Data Vault and Anchor methodologies, and walks through Alibaba’s multi‑stage data‑model practice, integration framework, dimension design, fact‑table strategies and aggregation techniques.
This guide walks you through the full Elasticsearch data‑modeling workflow, covering business‑driven design, handling large data volumes, optimal settings, mapping choices, and complex parent‑child relationships, while providing practical code examples and visual diagrams for immediate application.
This article chronicles the 30‑year dominance of relational databases, the challenges posed by object‑oriented and high‑concurrency workloads, and how four alternative data‑store families—Redis, MongoDB, Neo4j, and HBase/Cassandra—gave birth to the modern NoSQL movement.
This article explains what data architecture is, why it is essential for linking business and technology, outlines its main components such as data models, data flows, value streams and standards, and discusses emerging trends toward service‑oriented, consumption‑focused data architectures.
This article examines why databases are foundational to modern IT, traces the historical development of distributed database technologies, compares various architectural approaches such as sharding middleware, shared‑storage and shared‑nothing designs, and discusses emerging trends like multi‑model, HTAP, cloud‑native, and open‑source ecosystems.
This article examines the unique challenges of building B2B systems for a brand—covering multi‑role permissions, complex supply‑chain workflows, logistics integration, data modeling, and reliability—while sharing concrete architectural solutions such as cloud‑edge services, decision‑center IPC, end‑to‑end monitoring, and industry‑specific adaptations.