The article explains that, although MySQL’s theoretical limits are far higher, practical B+‑tree page and index calculations suggest keeping a single table under about twenty million one‑kilobyte rows to avoid performance degradation, illustrated with interview examples and estimation formulas.
This guide explains how data engineers can rapidly adapt to a new workplace by mastering business context, data domains, and system architecture, using structured learning, practical case studies, and continuous reflection to earn trust and deliver value efficiently.
This article compares MySQL and Elasticsearch across data models, query languages, indexing, distributed architecture, performance, scalability, and typical use cases, highlighting their distinct strengths and trade‑offs to help developers decide which system—or combination—best fits specific application requirements.
StarRocks 3.0 introduces a lake‑warehouse unified architecture where materialized views reduce data processing complexity, boost query performance, and improve timeliness, offering declarative modeling, transparent acceleration, and incremental computation for enterprise‑scale analytics.
This article compares SQL and NoSQL databases, outlining their advantages, limitations, and ideal use‑cases, and provides guidance on selecting the appropriate technology based on consistency, scalability, and data model requirements.
This article introduces graph databases as a NoSQL solution, explains property‑graph modeling, compares relational and document stores, evaluates several graph products, and details Dgraph’s architecture, indexing, query language, and real‑world business applications such as knowledge graphs and equity‑relationship analysis.
This article explains why data‑warehouse standards are essential for improving team efficiency, product quality, and maintenance costs, and provides a step‑by‑step guide covering standard creation, discussion, rollout, supervision, continuous improvement, as well as detailed design, process, quality, and security specifications.
The article presents Tencent Oula's comprehensive data‑asset platform that integrates data collection, integration, warehouse and metric modeling, unified services, and a governance engine to create trustworthy, low‑entropy data assets while addressing common data‑governance challenges and outlining future AI‑for‑BI possibilities.
This guide presents a comprehensive methodology for building knowledge graph schemas that decouple structural representation from semantic meaning, covering schema design, attribute semantic standardization, concept modeling, multi‑relational and hypergraph techniques, and practical steps for implementation across complex business domains.
This article presents a comprehensive methodology for building an Onedata data warehouse, covering the conceptual framework, data modeling processes, the Inmon and Kimball approaches, practical case studies from Baidu, Huawei, and banking, and key takeaways for enterprise data architecture.
Designing a time axis for HR systems involves choosing between an interval model that stores start and end timestamps and an effective‑date model that uses effective dates and sequences, each affecting query simplicity, maintenance effort, common pitfalls, and ultimately enabling accurate historical and future employee data analysis.
This guide walks through using the TuGraph graph database to design and deploy graph applications, covering real‑world scenarios, database selection, built‑in datasets, Docker/CentOS/Ubuntu deployment, model design, data import, debugging, operational monitoring, and integration with services or direct RESTful APIs.
The article traces the evolution from Turing's seminal AI test through the rise of ChatGPT, explains how large language models rely on knowledge graphs built from massive unstructured data, and examines the challenges and advantages of modern graph databases for high‑performance, flexible, and explainable AI applications.
The paper introduces SASS, a scenario-adaptive self-supervised recommendation model that uses contrastive pre-training and multi-layer gating to expand global samples and transfer scene-aware parameters, enabling a single model to deliver personalized recommendations across diverse Taobao ‘SuoSuo’ scenarios while mitigating data sparsity and cross-domain challenges.
This article explains data architecture and data modeling fundamentals, presents DAMA DMBOK concepts, outlines four practical solutions for model design, standard management, automated change control, and business mapping, and shares an enterprise manufacturing case study with Q&A on governance and efficiency.
This article presents a detailed roadmap for enterprise data governance, covering business digitization goals, data governance construction, typical digital platform architecture, core governance actions, implementation pathways, data asset inventory techniques, and real‑world case studies to illustrate practical execution.
This article outlines DeepEx Technology's comprehensive data governance methodology, covering construction paths, digital platform frameworks, core governance components, implementation steps, case studies, and a Q&A that together illustrate how enterprises can build reliable data assets, models, standards, and quality processes to unlock business value.
These COMAP MCM problem statements challenge teams to develop statistical models for pricing used sailboats using a large 2023 dataset and to propose innovative strategies for the Olympic Games, evaluating regional effects, data sources, and policy recommendations for sustainable hosting.
The article explains the purpose and benefits of data publication diagrams for visualizing relationships among data entities, services, and application components, introduces UML/BPMN EAP profile components with illustrative images, and outlines Archimate modeling elements, highlighting how data is localized within repositories or entity application components.
This article explains the concept of data models, why warehouse models need reconstruction, compares normative and dimensional modeling approaches, and provides a step‑by‑step guide—including information gathering, design, and implementation—to build efficient, maintainable data warehouse architectures.
This article explains what data models are, why model refactoring is necessary, compares normalized and dimensional data warehouse modeling approaches, and details a three‑step implementation process—including information research, model design, and model deployment—while highlighting best‑practice naming conventions and practical examples.
To handle exploding product inventory data, the company overhauled its warehouse inventory model by eliminating risky document‑hand‑offs, storing only changed rows instead of daily snapshots, and syncing transformed data to a data‑warehouse for reporting, which cut monthly accounting time by 30 hours (≈30 %), improved accuracy, enabled new analytics, and introduced TiDB migration and team upskilling.
This article details the challenges of fragmented ODS data in the moving‑service domain and explains how a dedicated public‑layer data warehouse, with layered architecture and quality monitoring, was designed and implemented to improve data reuse, reduce redundancy, and stabilize downstream analytics.
This article details Bilibili's 北极星 user behavior analysis platform, tracing its evolution from early Spark‑Jar models to Flink‑ClickHouse pipelines and Iceberg‑based full aggregation, and explains the technical solutions for event, retention, funnel, path analysis, data ingestion, cluster rebalancing, and performance optimizations that enable massive real‑time analytics on billions of daily events.
This comprehensive guide walks you through the entire data governance ecosystem, covering metadata fundamentals, classification, maturity models, data standards, modeling, integration, lifecycle management, quality assurance, security, and ETL processes, all illustrated with clear diagrams and practical steps.
This article outlines the design and implementation of a unified data service platform that standardizes data access, accelerates model processing, provides flexible API construction, and ensures high availability through gateway, caching, and monitoring, ultimately reducing cost and improving efficiency for both C‑end and B‑end applications.
The article outlines a standardized data‑service platform built atop a warehouse, detailing its construction, query, and gateway layers—supporting model definition, acceleration, reusable APIs, unified DSL/SQL interfaces, and observability—to solve ingestion, definition, and lineage issues, achieving 500+ APIs, sub‑day creation, and 18% cost reduction.
This article explains the concepts, architecture, and layered design of data warehouses, covering data flow, ETL processes, ODS, DWD, DWM, DWS, ADS layers, their characteristics, differences from databases, and the role of data marts in supporting OLAP and decision‑making.
This article explains the concept of Lifetime Value (LTV), presents a two‑step estimation method based on purchase pathways, evaluates model accuracy with real data, and discusses practical applications such as acquisition guidance, fine‑grained operations, and product iteration decisions.
This article details Xiaomi's multi‑year journey in building a group‑wide Business Intelligence platform, covering its historical evolution, technical challenges in performance, modeling, visualization and permissions, the current four‑layer architecture, and future plans to make the platform more business‑centric and simpler.
This comprehensive guide explains why data modeling is essential for big‑data systems, compares relational and OLAP approaches, details ER, dimensional, Data Vault and Anchor methodologies, and walks through Alibaba’s multi‑stage data‑model practice, integration framework, dimension design, fact‑table strategies and aggregation techniques.
This guide walks you through the full Elasticsearch data‑modeling workflow, covering business‑driven design, handling large data volumes, optimal settings, mapping choices, and complex parent‑child relationships, while providing practical code examples and visual diagrams for immediate application.
This article chronicles the 30‑year dominance of relational databases, the challenges posed by object‑oriented and high‑concurrency workloads, and how four alternative data‑store families—Redis, MongoDB, Neo4j, and HBase/Cassandra—gave birth to the modern NoSQL movement.
This article explains what data architecture is, why it is essential for linking business and technology, outlines its main components such as data models, data flows, value streams and standards, and discusses emerging trends toward service‑oriented, consumption‑focused data architectures.
This article examines why databases are foundational to modern IT, traces the historical development of distributed database technologies, compares various architectural approaches such as sharding middleware, shared‑storage and shared‑nothing designs, and discusses emerging trends like multi‑model, HTAP, cloud‑native, and open‑source ecosystems.
This article examines the unique challenges of building B2B systems for a brand—covering multi‑role permissions, complex supply‑chain workflows, logistics integration, data modeling, and reliability—while sharing concrete architectural solutions such as cloud‑edge services, decision‑center IPC, end‑to‑end monitoring, and industry‑specific adaptations.
This article details Kuaishou's end‑to‑end big data platform, describing its organizational model, unified data governance framework, comprehensive data‑quality solution, the design of a headless metric platform, key technologies such as automatic modeling and code generation, and future directions toward a decentralized, smart data fabric.
This article analyzes the limitations of traditional BI multi‑dimensional analysis that relies on wide tables and complex SQL JOINs, introduces a new DQL language that redefines JOIN operations into three plus one patterns, and demonstrates how DQL simplifies data modeling, reduces errors, and enables truly self‑service analytics.
This article analyzes Haro’s two‑wheeler dispatch problem, compares it with food‑delivery and ride‑hailing scheduling, outlines eight technical challenges, presents a multi‑fast‑good‑cheap algorithm framework, and details a simulation system that validates and improves the scheduling solution.
This article presents a comprehensive overview of Taobao's data model governance, detailing the background and problems of the current data architecture, analyzing root causes, proposing a structured governance framework with DataWorks automation, and outlining future plans to improve efficiency, standardization, and product tooling.
This article shares Fenbeitong’s practical experience building a big‑data platform for a young B2B company, covering company background, data‑team formation, technology selection, architecture design, governance processes, modeling tools, batch and real‑time modeling, and insights on ToB versus ToC technical choices.
This article examines the evolution of big‑data architectures, identifies the latency and maintenance issues of classic Lambda designs, and presents a hybrid Lambda‑Kappa solution that unifies streaming and batch processing to achieve minute‑level data freshness and second‑level query latency while reducing development cost.
This article revisits fundamental graph concepts, clarifies the distinction between graphs and charts, defines nodes, edges, geometry, and properties, and outlines data processing, layout strategies, and rendering approaches using Canvas and SVG, offering a concise roadmap for building a graph engine.
This article summarizes Guo Jinshi's presentation on Taobao's data model governance, covering the current data landscape, identified problems, analysis of root causes, proposed governance solutions—including DataWorks intelligent modeling—and future plans, while also providing a Q&A session on practical implementation.
This article explains JD's design of a billion‑scale vehicle‑part matching cache, detailing the data model, bitmap storage structure, SDS pre‑allocation, key‑value compression, cluster routing, and Lua‑based high‑performance search interface that achieves sub‑13 ms latency.
By abstracting 86 % of form‑list‑detail scenarios into standardized scene assets and linking them with unified data models, the platform enables no‑code generation of backend pages, cutting development effort fivefold versus source code, boosting overall productivity by 68.6 % while preserving product quality and consistency.
This article introduces Apache Doris, a high‑performance, real‑time analytical database built on an MPP architecture, covering its simple FE/BE design, three data‑model types (Aggregate, Unique, Duplicate), partitioning and bucketing strategies, rollup tables, and the limitations and best practices of row‑level updates.
This article introduces Apache IoTDB, an open‑source time‑series database for industrial IoT, outlines its recent feature releases, explains its data‑modeling and compression strategies, and discusses UDF, trigger, and quality‑control capabilities that guide technical selection and architecture design.
This document presents a comprehensive overview of the traffic domain in a data warehouse, covering its concepts, objectives, guiding principles, core and extension models, data quality, monitoring, scheduling, and operational practices to achieve a complete, accurate, efficient, low‑cost, and high‑value traffic data system while addressing massive data volume, consistency, and SLA challenges.
This article explains how different data models—from relational tables to JSON documents and graph structures—affect software design, storage, querying, and scalability, illustrating concepts with a resume example and discussing trade‑offs such as impedance mismatch, normalization, and multi‑entity relationships.
This article explains the variable clustering method, why it’s needed to reduce redundant variables, how to measure similarity using correlation coefficients or cosine angles, and describes common distance definitions such as maximum and minimum coefficient methods for effective factor selection.
This article provides a comprehensive guide to Apache Doris, covering how to create databases and tables with single and composite partitions, import data via broker and routine loads, understand its aggregate, uniq, and duplicate data models, and leverage rollup and prefix index features for optimized querying.
This article walks through the four essential stages of building a data project—data collection, modeling, analysis, and application—explaining key principles, common models such as 3NF, star/snowflake, cube, and wide tables, and comparing offline versus real‑time pipelines.
Taobao’s rapidly expanding technical data system faced naming inconsistencies, low table reuse, and costly, inefficient data usage, prompting a joint effort with DataWorks to digitize model evaluation, enforce standardized governance, deliver intelligent end‑to‑end modeling tools, and launch a development assistant, resulting in a health‑monitoring dashboard, upgraded data maps, and a roadmap for further automation and architecture refinement.
This article details Alibaba's large‑scale data model governance initiative, analyzing current data issues, presenting a comprehensive solution—including model digitization, public model sinking, productization, daily governance, and search‑enhancement—and outlining achieved results and future plans to further improve data quality, reuse, and operational efficiency.
This article explains how a scene‑driven, no‑code platform standardizes UI, data, and API models to enable scalable, high‑efficiency production of middle‑office pages, reduces manual coding, improves collaboration across roles, and provides a unified efficiency‑measurement framework for source, low‑code, and no‑code development modes.
This article explains ByteDance's data lineage architecture, covering data sources, processing pipelines, graph‑based modeling, key application scenarios, quality metrics such as accuracy, coverage and timeliness, and future directions for improving and standardizing lineage across its massive data platform.
This article walks through the step‑by‑step rewrite of the Dagoba in‑memory graph database from JavaScript to Python, covering data modeling, primary‑key management, eager and lazy query implementations, bidirectional edge support, performance optimizations, and how to extend the query language with custom methods.
This comprehensive guide explains Elasticsearch fundamentals—including documents, indices, nodes, clusters, REST and Document APIs, query DSL, relevance scoring, distributed architecture, real‑time indexing, search execution, pagination, scroll, aggregations, data modeling, mapping options, parent/child relationships, reindexing, and practical cluster and write/read performance optimizations.
This article explains how Kuashou tackled latency and efficiency problems in large‑scale data pipelines by adopting Apache Hudi, detailing the pain points, reasons for choosing Hudi, its architecture, model design, handling of bursty updates, back‑fill scenarios, and operational safeguards.
This comprehensive guide explains data‑warehouse construction standards, covering model architecture principles, layered design (ODS, DWD, DWM, DWS, APP), domain division, data‑model design rules, type and naming conventions, table lifecycle management, and practical implementation examples.
This article presents a step‑by‑step methodology for converting business domain models into robust data models, illustrated with a community group‑buying pre‑scheduling case, and discusses object identification, relationship definition, model refinement, and mapping to database structures to improve scalability and maintainability.
This article presents the OneData methodology for designing, standardizing, and governing a data warehouse, detailing background challenges, goals, industry references, core concepts, unified business and design consolidation, data modeling layers, naming conventions, data quality controls, and the resulting operational improvements and business value.
The article details Ctrip Business Travel's order system architecture, outlining its background, challenges of the original vertical‑only design, and the layered, process‑driven, and snapshot design principles that guide its evolution toward modular backend services, data consolidation, configurable messaging, and service orchestration.
This article presents a comprehensive guide to MySQL performance, covering maximum data volume and concurrency limits, recommended query response times, practical table‑design rules, index classification and optimization techniques, as well as detailed SQL tuning patterns such as batch processing, operator, OR, IN, LIKE, JOIN and LIMIT improvements.
The article explains the purpose of data class diagrams in TOGAF, describing how they model key business entities, their attributes and relationships at a high-level conceptual stage to aid understanding of the underlying data model, and includes examples from the tourism discount domain.
This article examines the challenges of evolving data models in large applications, explains the Command Query Responsibility Segregation (CQRS) pattern, and outlines when to adopt or avoid it to balance read/write performance, scalability, and system complexity.
This article explains the philosophical basis of object‑oriented thinking, distinguishes objects from things, discusses attributes and methods, compares object‑oriented and data‑model designs with Java and SQL examples, and clarifies composition and aggregation through real‑world and code illustrations.
This article explains what a data warehouse is, contrasts it with traditional databases, outlines how to design and build a data warehouse—including model selection, topic domain division, bus matrix, layered architecture, and data governance—then expands to the concept of a data middle platform and its distinction from data lakes and big‑data platforms.
This article explains what a data warehouse is, contrasts it with traditional databases, outlines how to design and build a warehouse—including model selection, subject‑area definition, bus matrix, layering, and data quality—while also covering related concepts such as data middle platforms, data lakes, metadata, and modeling techniques.
The article explains the rapid rise of graph databases, outlines their core concepts and advantages, compares them with NoSQL and relational databases, presents performance benchmarks, and reviews leading solutions such as Neo4j and JanusGraph, including their data models and query language.
This article explains the rise of graph databases, compares them with traditional relational and NoSQL systems, details the differences between Neo4j and JanusGraph, and demonstrates how the Cypher query language enables efficient relationship queries in complex, large‑scale data environments.
To meet Baidu Chinese’s need for sub‑200 ms responses on multi‑hop queries across millions of dictionary entities, the team replaced MySQL with the open‑source HugeGraph graph database backed by RocksDB, deploying a multi‑master, REST‑enabled architecture with caching, bulk loading, and a data‑intervention platform to ensure fast, reliable traversal of semantic relationships.
This article introduces graph fundamentals, various application scenarios such as science, code logic, Spark workflows, social networks, and event graphs, then details graph data modeling, analysis, matrix computations, and the deployment of graph neural networks using frameworks like DGL, highlighting practical engineering considerations.
This article details the historical development, architectural layers, ETL migration to Spark, data modeling standards, governance processes, resource optimization, security measures, and future roadmap of Meituan Waimai's offline data warehouse, illustrating how the team addressed scalability and efficiency challenges.
This article explains the fundamentals of ETL, describes data‑warehouse architectures such as star and snowflake schemas, outlines a five‑step enterprise‑level ETL workflow, and discusses advanced techniques, tools, and algorithms for building robust data‑warehouse solutions.
The article explains the concept of a data middle platform, detailing its three-layer structure—data model, data service, and data development—illustrates how data modeling enables cross-domain integration, how services encapsulate data for flexible consumption, and how development tools support customized data applications, using a telecom operator example.
This article details the background, challenges, and step‑by‑step reconstruction of Cloud Music's membership automation data warehouse, covering modeling standards, task decoupling, Spark performance tuning, and data quality checks to achieve higher efficiency and reliability.
This article provides a comprehensive overview of data warehouses, explaining their purpose, differences from databases, OLTP vs OLAP, traditional versus internet data warehouse models, layered architecture, modeling theories, metric dictionaries, date dimensions, naming conventions, data governance, and incremental synchronization techniques with practical SQL examples.
This article presents a comprehensive guide to data model architecture, detailing layered data store definitions, classification structures, processing flow, naming conventions, and core design principles to help engineers build scalable, maintainable data warehouses.
This article outlines a comprehensive data model architecture, detailing layered data stores, classification structures, naming conventions, and core modeling principles to guide effective data warehouse design and implementation.
This article presents a systematic, data‑model‑driven approach to Kubernetes stability assurance, detailing the sources of complexity, a four‑diagram and three‑table data model, insight and pre‑plan structures, global visualisation concepts, deployment patterns, operational workflows, and competitive analysis to enable effective, iterative, and sustainable cluster stability management.
The article explains the CQRS (Command Query Responsibility Segregation) pattern, showing how separating command and query models addresses diverse data view needs, improves query flexibility, discusses synchronization methods, compares with microservices and caching, and outlines its benefits and trade‑offs.
This article explains the concepts of data model, data dictionary, database schema, and ER diagram, clarifies their differences, and compares them across stages, purposes, detail levels, authors, users, tools, and forms to help readers grasp database design terminology.
This article provides a comprehensive guide to data model architecture, detailing layered data storage concepts, classification structures, naming conventions, and design principles to help practitioners build consistent, maintainable, and performant data warehouses and analytics platforms.
This article presents a comprehensive case study on how an education company leverages AI‑driven data modeling and multi‑source geographic and demographic data to optimize offline advertising placement within its OMO strategy, detailing background, methodology, technical implementation, challenges, and future outlook.
The article details the design and engineering of a high‑performance path analysis model for internet data operations, explaining session handling, Sankey visualizations, adjacency‑table storage, multi‑granular session partitioning, Spark‑to‑ClickHouse pipelines, and optimizations that enable billion‑scale user‑path queries in about one second.
This article outlines how a bank’s DBA team transformed traditional, slow, and error‑prone database release processes into a fully automated DevOps pipeline, covering modeling tools, the DBgo release engine, monitoring, AI‑ops, and practical lessons from real‑world deployments.
This tutorial explains how BPMN can be used to model data objects, groups, and text annotations within a business process, demonstrating the technique through a step‑by‑step case study of the True Aqua distilled‑water ordering workflow, including state management and final diagram creation.
This article describes how JD's Energy Management Platform leverages ClickHouse as a high‑performance, MPP‑based OLAP engine to provide real‑time, multi‑dimensional analytics on IoT energy data, covering business background, technology selection, system architecture, data ingestion, storage, replication, and a generic query interface with code examples.
This article explains the limitations of traditional REST APIs, introduces GraphQL as a flexible alternative, details its core concepts such as schema, types, queries, mutations and subscriptions, and outlines various deployment architectures and implementation considerations for backend development.
This article explains why REST APIs have become cumbersome, introduces GraphQL as a more flexible alternative, describes its core concepts such as schema, types and modifiers, and outlines various server‑side and client‑side integration patterns and tooling options.
This article walks through the challenges of querying a 300‑billion‑row Hive table, analyzes why traditional partitioning, indexing, and bucketing fall short, and presents a practical solution that combines active‑user segmentation and a redesigned array‑based data model to cut query time from hours to minutes.
This article details iQIYI's transition from a fragmented Data Warehouse 1.0 to a unified, standardized Data Warehouse 2.0, covering layered architecture, dimension and metric design, modeling workflows, metadata management, data lineage, and upcoming intelligent and automated data platform initiatives.
This article explains how Alibaba's traffic analysis platform evolved to require real‑time analytics, outlines the business background, data model and metric system, discusses the difficulties of big‑data processing, and describes why ClickHouse was chosen and how its features solve those challenges.
This article explains how to use Redis sets, sorted sets, bitmaps, and HyperLogLog for various aggregation, sorting, binary state, and cardinality statistics, providing concrete command examples, design patterns, and performance considerations for large‑scale applications.
This article outlines a comprehensive framework for designing decision‑type products, covering their evolution stages, core elements of model‑data‑strategy, domain modeling techniques, data‑to‑knowledge transformation, business and process value, and a feedback‑driven decision loop with evaluation and simulation.
The article explains why relational databases, especially PostgreSQL combined with TimescaleDB, are better suited than NoSQL for handling complex, time‑series IoT data, offering robust SQL ecosystems, schema design, data integrity, and reduced data silos for multi‑user access.
The talk chronicles iQIYI’s shift from a fragmented five‑layer Data Warehouse 1.0 to a unified 2.0 architecture featuring a central Dimension Layer, business‑focused data marts, and subject‑oriented warehouses, while detailing platform services, rigorous metadata management, lineage tracking, and future goals of intelligent, automated, service‑oriented, model‑driven data governance.
The article explains how to improve Elasticsearch query speed on billions of records by leveraging filesystem cache, reducing indexed fields, using data pre‑heating, separating hot and cold indices, designing efficient document models, and applying pagination techniques such as scroll API and search_after.
