This article describes how a C++‑based cross‑platform framework and a unified routing mechanism were built for WeChat Pay to eliminate platform‑specific bugs, streamline data flow, improve crash stability, cut code size by nearly 45%, and boost development productivity across iOS and Android.
This article explores the complete lifecycle of an indicator system—from defining metrics and addressing common ambiguities, through designing concept consensus, semantic layers, mechanisms, and governance, to productizing platforms, optimizing development, and envisioning future AI‑driven enhancements.
This comprehensive guide explains the purpose of architecture diagrams, the criteria for good diagrams, the optimal moments to create them, and detailed methods for drawing business, application, technical, code, and data architecture diagrams, complete with design principles, classification, and practical tips.
This article explains how to build and apply a comprehensive metric system by addressing business, technical, and product challenges, outlining design, standardization, metadata management, and future AI‑driven use cases to support data‑driven decision making.
Grey Theory, introduced by Deng Julong in 1982, offers a mathematical framework for analyzing systems with incomplete or uncertain data, using techniques like generated series and the GM(1,1) model to enable reliable forecasting and decision‑making across fields such as economics, environment, and product lifecycle analysis.
The article explains retention analysis models, their importance for user loyalty, outlines offline Hive architecture, then shows how ClickHouse’s retention() function and columnar storage dramatically speed up multi‑day retention calculations, providing SQL examples and practical guidance for product analytics.
This article explains the concept of a metrics layer, why StarRocks is suited for building such platforms, and presents detailed case studies from Airbnb, a major bank, and a leading restaurant chain, while comparing three modeling paradigms and outlining the future vision for materialized views.
The article explains how to analyze, decompose, and design a payment system by breaking the workflow into modules, illustrating pre‑payment, third‑party integration, and post‑payment stages with sequence diagrams, and proposing a data‑structure model that covers account management, transaction records, order handling, and related business components.
This article outlines the securities sector's data‑governance background, vision, common pain points, and a comprehensive service‑oriented framework covering data models, quality, standards, metadata, and security, and demonstrates the resulting business value and future outlook.
This article explains how to design and implement a traffic distribution diagram for an observability system, covering current cloud‑native tooling, data standardization, transformation, traffic‑flow modeling, aggregation, storage with ClickHouse, and visualisation techniques such as Sankey diagrams.
This article explains how Heartbeat Game created a cloud‑native big data platform on Alibaba Cloud using DataWorks, detailing the end‑to‑end data pipeline, a universal logical data model for games, and the advantages of the GS‑LBDM architecture for AI, risk control, and analytics scenarios.
This talk by Ant Group’s senior technical expert Wang Gaohang details the definition, design, mechanism, productization, and future outlook of the company’s metric system, covering concept consensus, semantic layers, workflow, AI assistance, performance optimization, and practical case studies.
This article explains how agile methodologies can be applied to data management, covering the need for agile data practices, core principles, iterative modeling, governance, CI/CD pipelines, tooling, metrics, security, case studies, challenges, and future outlooks in a comprehensive, step‑by‑step guide.
Discord’s rapid growth forced a series of massive database migrations—from MongoDB to Cassandra in 2017, then to ScyllaDB in 2023—detailing the motivations, requirements, data‑model design, performance challenges, migration tooling, and the resulting operational improvements.
This article presents a comprehensive overview of Xiaomi's sales data warehouse, covering its evolution, dimensional modeling and layer theory, Lambda architecture with batch and streaming processing, capability layers, security measures, and future trends toward real‑time metricization and data value creation.
This article details the background, challenges, design principles, and implementation of a unified data service platform—including data modeling, multi-source governance, real-time processing, and a Doris-based storage solution—to support large‑scale video data for a new media operation.
This article explains how to redesign a Java data model so that a single sessionId is stored at the outer level while multiple sceneId/sceneName pairs are nested inside a sceneList, and shows the required MyBatis mapper, XML mapping, service, and controller implementations.
This article explains what NoSQL databases are, outlines their typical use cases and architectural components, compares them with traditional relational databases across storage, scalability, query, and transaction aspects, and highlights the advantages and trade‑offs to consider when choosing a data solution.
This article details NetEase EasyData's evolution and product matrix, explains why a metric middle platform is needed, describes its core technical architecture—including a unified logical semantic model, a custom metric query language, and engine decoupling—and outlines future development directions.
The article details OPPO’s approach to balancing cost reduction and user value in app distribution through unified cross‑scenario modeling, sparse‑data solutions, oCPX advertising optimization, and a multi‑stage smart‑power system that improves efficiency, scalability, and revenue while preserving user experience.
The article explains the differences between data schemas, data structures, and data models, proposes a systematic naming approach, and outlines a five‑schema architecture—including business, view, logical, deployment, and physical schemas—while addressing terminology challenges and normalization processes.
This article introduces NoSQL databases, explains their core concepts, typical use cases, architectural components, and contrasts them with relational databases, helping readers understand when and why to choose NoSQL solutions for large‑scale, unstructured data workloads.
The article details JD Logistics' order system redesign using a four‑layer transaction architecture, describing its decoupled backend, unified data model, high‑availability components such as CQRS, Redis, JMQ, HBase, and Elasticsearch, and outlines design advantages, extensible data modeling, future challenges, and overall performance outcomes.
This article introduces Alibaba Cloud DataWorks' intelligent data modeling tool, outlines the data demand flow, shares best practices and hands‑on demonstrations for data warehouse modeling, discusses common challenges and their solutions, and provides Q&A and product details for developers and data engineers.
This article explains what a data warehouse is, contrasts it with traditional databases, outlines the evolution from classic to internet‑scale warehouses, details modeling approaches and layered architectures, discusses KPI dictionaries, date dimensions, naming standards, data governance, incremental loading techniques, and upstream/downstream coordination.
This article explains how to optimize OLAP workloads by balancing normalization and denormalization, applying data sharding, replication, indexing, partitioning, materialized views, columnar storage, compression, and lifecycle management, and provides practical ClickHouse SQL examples for index creation, partitioning, and query plan analysis.
This article outlines how a marketing‑focused data middle platform can be constructed by integrating online and offline behavior data, business data, and third‑party sources, then applying data integration, modeling, processing, and application capabilities to enable data‑driven user journeys and personalized marketing strategies.
This article walks through a real‑world e‑commerce scenario where billions of daily browsing events generate over 100K TPS writes, evaluates storage options on reliability, cost, read/write performance and implementation difficulty, and ultimately recommends Tair after detailed analysis of List, Sorted‑Set and Hash structures, code examples, and concurrency controls.
This article details the design of a decoupled, high‑availability order system, covering business scope, value propositions, layered architecture, real‑time data layer, read/write separation, caching, messaging, search, multi‑tenant support, data security, and future challenges such as personalized queries and cost‑effective scaling.
The article explains that, although MySQL’s theoretical limits are far higher, practical B+‑tree page and index calculations suggest keeping a single table under about twenty million one‑kilobyte rows to avoid performance degradation, illustrated with interview examples and estimation formulas.
This guide explains how data engineers can rapidly adapt to a new workplace by mastering business context, data domains, and system architecture, using structured learning, practical case studies, and continuous reflection to earn trust and deliver value efficiently.
This article compares MySQL and Elasticsearch across data models, query languages, indexing, distributed architecture, performance, scalability, and typical use cases, highlighting their distinct strengths and trade‑offs to help developers decide which system—or combination—best fits specific application requirements.
StarRocks 3.0 introduces a lake‑warehouse unified architecture where materialized views reduce data processing complexity, boost query performance, and improve timeliness, offering declarative modeling, transparent acceleration, and incremental computation for enterprise‑scale analytics.
This article compares SQL and NoSQL databases, outlining their advantages, limitations, and ideal use‑cases, and provides guidance on selecting the appropriate technology based on consistency, scalability, and data model requirements.
This article introduces graph databases as a NoSQL solution, explains property‑graph modeling, compares relational and document stores, evaluates several graph products, and details Dgraph’s architecture, indexing, query language, and real‑world business applications such as knowledge graphs and equity‑relationship analysis.
This article explains why data‑warehouse standards are essential for improving team efficiency, product quality, and maintenance costs, and provides a step‑by‑step guide covering standard creation, discussion, rollout, supervision, continuous improvement, as well as detailed design, process, quality, and security specifications.
The article presents Tencent Oula's comprehensive data‑asset platform that integrates data collection, integration, warehouse and metric modeling, unified services, and a governance engine to create trustworthy, low‑entropy data assets while addressing common data‑governance challenges and outlining future AI‑for‑BI possibilities.
This guide presents a comprehensive methodology for building knowledge graph schemas that decouple structural representation from semantic meaning, covering schema design, attribute semantic standardization, concept modeling, multi‑relational and hypergraph techniques, and practical steps for implementation across complex business domains.
This article presents a comprehensive methodology for building an Onedata data warehouse, covering the conceptual framework, data modeling processes, the Inmon and Kimball approaches, practical case studies from Baidu, Huawei, and banking, and key takeaways for enterprise data architecture.
Designing a time axis for HR systems involves choosing between an interval model that stores start and end timestamps and an effective‑date model that uses effective dates and sequences, each affecting query simplicity, maintenance effort, common pitfalls, and ultimately enabling accurate historical and future employee data analysis.
This guide walks through using the TuGraph graph database to design and deploy graph applications, covering real‑world scenarios, database selection, built‑in datasets, Docker/CentOS/Ubuntu deployment, model design, data import, debugging, operational monitoring, and integration with services or direct RESTful APIs.
The article traces the evolution from Turing's seminal AI test through the rise of ChatGPT, explains how large language models rely on knowledge graphs built from massive unstructured data, and examines the challenges and advantages of modern graph databases for high‑performance, flexible, and explainable AI applications.
The paper introduces SASS, a scenario-adaptive self-supervised recommendation model that uses contrastive pre-training and multi-layer gating to expand global samples and transfer scene-aware parameters, enabling a single model to deliver personalized recommendations across diverse Taobao ‘SuoSuo’ scenarios while mitigating data sparsity and cross-domain challenges.
This article explains data architecture and data modeling fundamentals, presents DAMA DMBOK concepts, outlines four practical solutions for model design, standard management, automated change control, and business mapping, and shares an enterprise manufacturing case study with Q&A on governance and efficiency.
This article presents a detailed roadmap for enterprise data governance, covering business digitization goals, data governance construction, typical digital platform architecture, core governance actions, implementation pathways, data asset inventory techniques, and real‑world case studies to illustrate practical execution.
This article outlines DeepEx Technology's comprehensive data governance methodology, covering construction paths, digital platform frameworks, core governance components, implementation steps, case studies, and a Q&A that together illustrate how enterprises can build reliable data assets, models, standards, and quality processes to unlock business value.
These COMAP MCM problem statements challenge teams to develop statistical models for pricing used sailboats using a large 2023 dataset and to propose innovative strategies for the Olympic Games, evaluating regional effects, data sources, and policy recommendations for sustainable hosting.
The article explains the purpose and benefits of data publication diagrams for visualizing relationships among data entities, services, and application components, introduces UML/BPMN EAP profile components with illustrative images, and outlines Archimate modeling elements, highlighting how data is localized within repositories or entity application components.
This article explains the concept of data models, why warehouse models need reconstruction, compares normative and dimensional modeling approaches, and provides a step‑by‑step guide—including information gathering, design, and implementation—to build efficient, maintainable data warehouse architectures.
This article explains what data models are, why model refactoring is necessary, compares normalized and dimensional data warehouse modeling approaches, and details a three‑step implementation process—including information research, model design, and model deployment—while highlighting best‑practice naming conventions and practical examples.
To handle exploding product inventory data, the company overhauled its warehouse inventory model by eliminating risky document‑hand‑offs, storing only changed rows instead of daily snapshots, and syncing transformed data to a data‑warehouse for reporting, which cut monthly accounting time by 30 hours (≈30 %), improved accuracy, enabled new analytics, and introduced TiDB migration and team upskilling.
This article details the challenges of fragmented ODS data in the moving‑service domain and explains how a dedicated public‑layer data warehouse, with layered architecture and quality monitoring, was designed and implemented to improve data reuse, reduce redundancy, and stabilize downstream analytics.
This article details Bilibili's 北极星 user behavior analysis platform, tracing its evolution from early Spark‑Jar models to Flink‑ClickHouse pipelines and Iceberg‑based full aggregation, and explains the technical solutions for event, retention, funnel, path analysis, data ingestion, cluster rebalancing, and performance optimizations that enable massive real‑time analytics on billions of daily events.
This comprehensive guide walks you through the entire data governance ecosystem, covering metadata fundamentals, classification, maturity models, data standards, modeling, integration, lifecycle management, quality assurance, security, and ETL processes, all illustrated with clear diagrams and practical steps.
This article outlines the design and implementation of a unified data service platform that standardizes data access, accelerates model processing, provides flexible API construction, and ensures high availability through gateway, caching, and monitoring, ultimately reducing cost and improving efficiency for both C‑end and B‑end applications.
The article outlines a standardized data‑service platform built atop a warehouse, detailing its construction, query, and gateway layers—supporting model definition, acceleration, reusable APIs, unified DSL/SQL interfaces, and observability—to solve ingestion, definition, and lineage issues, achieving 500+ APIs, sub‑day creation, and 18% cost reduction.
This article explains the concepts, architecture, and layered design of data warehouses, covering data flow, ETL processes, ODS, DWD, DWM, DWS, ADS layers, their characteristics, differences from databases, and the role of data marts in supporting OLAP and decision‑making.
This article explains the concept of Lifetime Value (LTV), presents a two‑step estimation method based on purchase pathways, evaluates model accuracy with real data, and discusses practical applications such as acquisition guidance, fine‑grained operations, and product iteration decisions.
This article details Xiaomi's multi‑year journey in building a group‑wide Business Intelligence platform, covering its historical evolution, technical challenges in performance, modeling, visualization and permissions, the current four‑layer architecture, and future plans to make the platform more business‑centric and simpler.
This comprehensive guide explains why data modeling is essential for big‑data systems, compares relational and OLAP approaches, details ER, dimensional, Data Vault and Anchor methodologies, and walks through Alibaba’s multi‑stage data‑model practice, integration framework, dimension design, fact‑table strategies and aggregation techniques.
This guide walks you through the full Elasticsearch data‑modeling workflow, covering business‑driven design, handling large data volumes, optimal settings, mapping choices, and complex parent‑child relationships, while providing practical code examples and visual diagrams for immediate application.
This article chronicles the 30‑year dominance of relational databases, the challenges posed by object‑oriented and high‑concurrency workloads, and how four alternative data‑store families—Redis, MongoDB, Neo4j, and HBase/Cassandra—gave birth to the modern NoSQL movement.
This article explains what data architecture is, why it is essential for linking business and technology, outlines its main components such as data models, data flows, value streams and standards, and discusses emerging trends toward service‑oriented, consumption‑focused data architectures.
This article examines why databases are foundational to modern IT, traces the historical development of distributed database technologies, compares various architectural approaches such as sharding middleware, shared‑storage and shared‑nothing designs, and discusses emerging trends like multi‑model, HTAP, cloud‑native, and open‑source ecosystems.
This article examines the unique challenges of building B2B systems for a brand—covering multi‑role permissions, complex supply‑chain workflows, logistics integration, data modeling, and reliability—while sharing concrete architectural solutions such as cloud‑edge services, decision‑center IPC, end‑to‑end monitoring, and industry‑specific adaptations.
This article details Kuaishou's end‑to‑end big data platform, describing its organizational model, unified data governance framework, comprehensive data‑quality solution, the design of a headless metric platform, key technologies such as automatic modeling and code generation, and future directions toward a decentralized, smart data fabric.
This article analyzes the limitations of traditional BI multi‑dimensional analysis that relies on wide tables and complex SQL JOINs, introduces a new DQL language that redefines JOIN operations into three plus one patterns, and demonstrates how DQL simplifies data modeling, reduces errors, and enables truly self‑service analytics.
This article analyzes Haro’s two‑wheeler dispatch problem, compares it with food‑delivery and ride‑hailing scheduling, outlines eight technical challenges, presents a multi‑fast‑good‑cheap algorithm framework, and details a simulation system that validates and improves the scheduling solution.
This article presents a comprehensive overview of Taobao's data model governance, detailing the background and problems of the current data architecture, analyzing root causes, proposing a structured governance framework with DataWorks automation, and outlining future plans to improve efficiency, standardization, and product tooling.
This article shares Fenbeitong’s practical experience building a big‑data platform for a young B2B company, covering company background, data‑team formation, technology selection, architecture design, governance processes, modeling tools, batch and real‑time modeling, and insights on ToB versus ToC technical choices.
This article examines the evolution of big‑data architectures, identifies the latency and maintenance issues of classic Lambda designs, and presents a hybrid Lambda‑Kappa solution that unifies streaming and batch processing to achieve minute‑level data freshness and second‑level query latency while reducing development cost.
This article revisits fundamental graph concepts, clarifies the distinction between graphs and charts, defines nodes, edges, geometry, and properties, and outlines data processing, layout strategies, and rendering approaches using Canvas and SVG, offering a concise roadmap for building a graph engine.
This article summarizes Guo Jinshi's presentation on Taobao's data model governance, covering the current data landscape, identified problems, analysis of root causes, proposed governance solutions—including DataWorks intelligent modeling—and future plans, while also providing a Q&A session on practical implementation.
This article explains JD's design of a billion‑scale vehicle‑part matching cache, detailing the data model, bitmap storage structure, SDS pre‑allocation, key‑value compression, cluster routing, and Lua‑based high‑performance search interface that achieves sub‑13 ms latency.
By abstracting 86 % of form‑list‑detail scenarios into standardized scene assets and linking them with unified data models, the platform enables no‑code generation of backend pages, cutting development effort fivefold versus source code, boosting overall productivity by 68.6 % while preserving product quality and consistency.
This article introduces Apache Doris, a high‑performance, real‑time analytical database built on an MPP architecture, covering its simple FE/BE design, three data‑model types (Aggregate, Unique, Duplicate), partitioning and bucketing strategies, rollup tables, and the limitations and best practices of row‑level updates.
This article introduces Apache IoTDB, an open‑source time‑series database for industrial IoT, outlines its recent feature releases, explains its data‑modeling and compression strategies, and discusses UDF, trigger, and quality‑control capabilities that guide technical selection and architecture design.
This document presents a comprehensive overview of the traffic domain in a data warehouse, covering its concepts, objectives, guiding principles, core and extension models, data quality, monitoring, scheduling, and operational practices to achieve a complete, accurate, efficient, low‑cost, and high‑value traffic data system while addressing massive data volume, consistency, and SLA challenges.
This article explains how different data models—from relational tables to JSON documents and graph structures—affect software design, storage, querying, and scalability, illustrating concepts with a resume example and discussing trade‑offs such as impedance mismatch, normalization, and multi‑entity relationships.
This article explains the variable clustering method, why it’s needed to reduce redundant variables, how to measure similarity using correlation coefficients or cosine angles, and describes common distance definitions such as maximum and minimum coefficient methods for effective factor selection.
This article provides a comprehensive guide to Apache Doris, covering how to create databases and tables with single and composite partitions, import data via broker and routine loads, understand its aggregate, uniq, and duplicate data models, and leverage rollup and prefix index features for optimized querying.
This article walks through the four essential stages of building a data project—data collection, modeling, analysis, and application—explaining key principles, common models such as 3NF, star/snowflake, cube, and wide tables, and comparing offline versus real‑time pipelines.
Taobao’s rapidly expanding technical data system faced naming inconsistencies, low table reuse, and costly, inefficient data usage, prompting a joint effort with DataWorks to digitize model evaluation, enforce standardized governance, deliver intelligent end‑to‑end modeling tools, and launch a development assistant, resulting in a health‑monitoring dashboard, upgraded data maps, and a roadmap for further automation and architecture refinement.
This article details Alibaba's large‑scale data model governance initiative, analyzing current data issues, presenting a comprehensive solution—including model digitization, public model sinking, productization, daily governance, and search‑enhancement—and outlining achieved results and future plans to further improve data quality, reuse, and operational efficiency.
This article explains how a scene‑driven, no‑code platform standardizes UI, data, and API models to enable scalable, high‑efficiency production of middle‑office pages, reduces manual coding, improves collaboration across roles, and provides a unified efficiency‑measurement framework for source, low‑code, and no‑code development modes.
This article explains ByteDance's data lineage architecture, covering data sources, processing pipelines, graph‑based modeling, key application scenarios, quality metrics such as accuracy, coverage and timeliness, and future directions for improving and standardizing lineage across its massive data platform.
This article walks through the step‑by‑step rewrite of the Dagoba in‑memory graph database from JavaScript to Python, covering data modeling, primary‑key management, eager and lazy query implementations, bidirectional edge support, performance optimizations, and how to extend the query language with custom methods.
This comprehensive guide explains Elasticsearch fundamentals—including documents, indices, nodes, clusters, REST and Document APIs, query DSL, relevance scoring, distributed architecture, real‑time indexing, search execution, pagination, scroll, aggregations, data modeling, mapping options, parent/child relationships, reindexing, and practical cluster and write/read performance optimizations.
This article explains how Kuashou tackled latency and efficiency problems in large‑scale data pipelines by adopting Apache Hudi, detailing the pain points, reasons for choosing Hudi, its architecture, model design, handling of bursty updates, back‑fill scenarios, and operational safeguards.
This comprehensive guide explains data‑warehouse construction standards, covering model architecture principles, layered design (ODS, DWD, DWM, DWS, APP), domain division, data‑model design rules, type and naming conventions, table lifecycle management, and practical implementation examples.
This article presents a step‑by‑step methodology for converting business domain models into robust data models, illustrated with a community group‑buying pre‑scheduling case, and discusses object identification, relationship definition, model refinement, and mapping to database structures to improve scalability and maintainability.
This article presents the OneData methodology for designing, standardizing, and governing a data warehouse, detailing background challenges, goals, industry references, core concepts, unified business and design consolidation, data modeling layers, naming conventions, data quality controls, and the resulting operational improvements and business value.
This guide walks through discovering Snuba datasets and entities, inspecting their schemas, building queries with the Python SDK or Sentry code, testing via the Web UI or curl, and interpreting request, response, and debugging information including HTTP status codes and ClickHouse metrics.
The article details Ctrip Business Travel's order system architecture, outlining its background, challenges of the original vertical‑only design, and the layered, process‑driven, and snapshot design principles that guide its evolution toward modular backend services, data consolidation, configurable messaging, and service orchestration.
This article presents a comprehensive guide to MySQL performance, covering maximum data volume and concurrency limits, recommended query response times, practical table‑design rules, index classification and optimization techniques, as well as detailed SQL tuning patterns such as batch processing, operator, OR, IN, LIKE, JOIN and LIMIT improvements.
The article explains the purpose of data class diagrams in TOGAF, describing how they model key business entities, their attributes and relationships at a high-level conceptual stage to aid understanding of the underlying data model, and includes examples from the tourism discount domain.
This article examines the challenges of evolving data models in large applications, explains the Command Query Responsibility Segregation (CQRS) pattern, and outlines when to adopt or avoid it to balance read/write performance, scalability, and system complexity.
The article explains Snuba’s data organization by separating a logical model visible to clients from a physical model that maps to ClickHouse tables, detailing datasets, entity types, relationships, consistency guarantees, storage layers, and concrete examples of single and multi‑entity datasets.
This article explains the philosophical basis of object‑oriented thinking, distinguishes objects from things, discusses attributes and methods, compares object‑oriented and data‑model designs with Java and SQL examples, and clarifies composition and aggregation through real‑world and code illustrations.
This article explains what a data warehouse is, contrasts it with traditional databases, outlines how to design and build a data warehouse—including model selection, topic domain division, bus matrix, layered architecture, and data governance—then expands to the concept of a data middle platform and its distinction from data lakes and big‑data platforms.
