The article shares a fresh graduate's experience interviewing for offline data‑warehouse roles at companies like Ctrip, Meituan and Alibaba, outlines the common interview pattern, and lists detailed Spark, Hadoop, and data‑warehouse questions used by these firms.
This article presents a comprehensive methodology for building an Onedata data warehouse, covering the conceptual framework, data modeling processes, the Inmon and Kimball approaches, practical case studies from Baidu, Huawei, and banking, and key takeaways for enterprise data architecture.
This article examines the explosion of heterogeneous data sources, the limitations of traditional data lakes and warehouses, and proposes a distributed lakehouse architecture that integrates advanced management layers to improve data governance, reliability, and support both SQL and advanced analytics workloads.
This presentation introduces Apache Doris, examines changing data analysis demands in the cloud era, explains why SelectDB was created, and details SelectDB’s cloud‑native architecture, performance, unified capabilities, ease of use, cost efficiency, open‑source nature, and its application scenarios for modern data warehousing and log analytics.
This article details Tongcheng Travel’s production deployment of the StarRocks OLAP database, covering background, business scenarios, technical evaluation against ClickHouse and Greenplum, implementation with Flink SQL, real‑time analytics, offline reporting, CDP use cases, performance optimizations, and future cloud‑native plans.
This article details Waterdrop's transition from ClickHouse and TiDB to a unified StarRocks‑based OLAP platform on Alibaba Cloud EMR, covering architecture evolution, performance benchmarks, best‑practice recommendations, and future plans for large‑scale real‑time analytics.
LakeSoul, China's only open‑source lakehouse project, has been donated to the LF AI & Data Foundation, becoming its first lake‑warehouse framework and offering ACID‑guaranteed high‑concurrency upserts, a high‑performance Rust‑based I/O layer, real‑time data‑warehouse capabilities, and seamless AI/BI integration for modern big‑data applications.
This article explains how building hash cluster tables in MaxCompute can compress pre‑sorted data, enable shuffle removal, and dramatically reduce execution time and resource consumption for conversion attribution tasks.
This article details Dingdong Maicai's adoption of Apache Doris as a real‑time OLAP engine, covering business requirements, comparative evaluation with ClickHouse, system architecture, practical applications such as real‑time analytics, B‑end queries, tag systems, and performance‑boosting techniques like Colocate Join, bitmap, prefix and Bloom‑filter indexes, materialized views, and streamlined Broker Load workflows.
This article explains why lake‑warehouse integration is needed, outlines its challenges, describes StarRocks' four integration paradigms—including query acceleration, layered modeling, real‑time warehouse‑lake fusion, and the cloud‑native 3.0 solution—and previews the upcoming StarRocks 3.0 release.
This article details how Hangzhou Bank Consumer Finance modernized its big‑data platform by introducing Apache Doris 1.2, replacing the original Greenplum + CDH architecture, unifying data sources via Multi‑Catalog, achieving second‑level query latency, reducing storage and compute costs, and outlining the integration workflow with DolphinScheduler, SeaTunnel, and Spark.
This article details Bilibili’s migration from a Spark‑based offline ODS‑to‑DWD sharding process to a Flink real‑time incremental pipeline, explaining the background challenges, the design of multi‑level partitioning, small‑file optimizations, stability enhancements, and the measurable performance gains achieved.
This article explains the concepts, key characteristics, terminology, and practical steps of dimensional modeling—including star and snowflake schemas—and demonstrates how to apply the methodology to a real‑world sales analysis scenario, while also discussing common challenges in building star‑schema models.
This article details the migration from a traditional Hive‑based data warehouse to a lakehouse architecture using Apache Hudi, covering the original Lambda setup, its pain points, lake‑vs‑warehouse differences, Hudi features, integration challenges, practical solutions, and future roadmap.
This article reviews the origins, core characteristics, traditional and logical architectures of data warehouses, explores emerging trends such as massive real‑time data, and outlines Huawei Cloud GaussDB(DWS) evolution toward a cloud‑native, elastic, lake‑warehouse integrated solution.
The article details Baidu's precise watermark design for its unified streaming‑batch data warehouse, describing how a centralized watermark server and client ensure end‑to‑end data completeness, align real‑time and batch windows with 99.9‑99.99% precision, and support accurate anti‑fraud calculations within the broader big‑data ecosystem.
Apache Flink 1.17 introduces a suite of batch and streaming enhancements—including a new Streaming Warehouse API, significant TPC‑DS performance boosts, adaptive batch scheduling, improved checkpointing, expanded SQL capabilities, Hive connector upgrades, and broader filesystem support—while also delivering upgrades to FRocksDB, Calcite, and the token framework to strengthen its position as a leading unified data‑processing engine.
This article explains how Inceptor and Apache Delta Lake combine distributed transaction, MVCC, snapshot isolation, and high‑performance SQL to support both data lake and data warehouse workloads, compares them with Hudi and Iceberg, and outlines their strengths and limitations for modern big‑data analytics.
This article explains how the lakehouse integrated architecture combines data lake and data warehouse capabilities, outlines its key features, compares three implementation paths, and provides an in‑depth technical overview of Apache Hudi and Apache Iceberg for modern big‑data analytics.
This article explains how Apache Doris extends its real‑time OLAP capabilities to support Lakehouse architectures, covering unified metadata, query acceleration, elastic compute, performance benchmarks, and future roadmap for richer data‑source integration and resource isolation.
This article traces 20 years of big‑data evolution, compares data lakes and data warehouses, defines both concepts, examines their technical trade‑offs, and presents Alibaba Cloud’s lake‑warehouse (lakehouse) solution that unifies flexible storage with enterprise‑grade performance and governance.
This article traces a decade of Amazon Redshift’s evolution, detailing its shift from a traditional MPP warehouse to a fully cloud‑native Serverless architecture, exploring its underlying innovations, key features such as Concurrency Scaling, built‑in ML, Data Sharing, and offering practical best‑practice guidance for real‑time analytics across diverse industry scenarios.
This article explains the concept of data models, why warehouse models need reconstruction, compares normative and dimensional modeling approaches, and provides a step‑by‑step guide—including information gathering, design, and implementation—to build efficient, maintainable data warehouse architectures.
The article analyzes the growing performance challenges of data warehouses, evaluates traditional solutions such as clustering, pre‑computation and optimization engines, and presents esProc SPL as a non‑SQL, low‑complexity alternative that delivers orders‑of‑magnitude speedups on modest hardware.
This article provides an overview of the most popular ETL tools—both open‑source and commercial—explaining their core features, use cases, and how they simplify data extraction, transformation, and loading for modern data‑driven applications.
This article explains what data models are, why model refactoring is necessary, compares normalized and dimensional data warehouse modeling approaches, and details a three‑step implementation process—including information research, model design, and model deployment—while highlighting best‑practice naming conventions and practical examples.
To handle exploding product inventory data, the company overhauled its warehouse inventory model by eliminating risky document‑hand‑offs, storing only changed rows instead of daily snapshots, and syncing transformed data to a data‑warehouse for reporting, which cut monthly accounting time by 30 hours (≈30 %), improved accuracy, enabled new analytics, and introduced TiDB migration and team upskilling.
This article introduces the concept of ETL, explains its importance for modern data‑driven applications, and provides a comprehensive comparison of the most popular open‑source and commercial ETL platforms—including their key features, supported data sources, and deployment options—helping readers choose the right tool for their data integration needs.
This article details the challenges of fragmented ODS data in the moving‑service domain and explains how a dedicated public‑layer data warehouse, with layered architecture and quality monitoring, was designed and implemented to improve data reuse, reduce redundancy, and stabilize downstream analytics.
This article details the evolution of Tencent Music's content library data platform from version 1.0 to 4.0, describing business requirements, architectural redesigns—including migration from ClickHouse to Apache Doris, introduction of a semantic layer, and extensive write, query, and cost optimizations—while sharing practical lessons and future directions.
This article summarizes Wang Chuanting’s DTCC2022 talk on Huawei Cloud GaussDB(DWS) 3.0, detailing its cloud‑native architecture, layered elasticity, lake‑warehouse integration, performance acceleration techniques, and how it seamlessly couples data‑processing pipelines with AI workloads for modern, real‑time analytics.
The article analyzes Teradata's withdrawal from the Chinese market, examines its shared‑nothing MPP architecture, compares mainstream domestic data‑warehouse designs, reviews global Gartner assessments, and highlights local alternatives such as GBase 8a and Huawei GaussDB DWS along with migration tools.
Yidian Tianxia tackled massive daily data volumes and complex analytics by defining a five‑layer data‑warehouse standard, comparing ClickHouse and StarRocks performance, and implementing a unified real‑time/offline architecture with StarRocks, DataPlus, and EasyJob, achieving multi‑fold query speedups and lower operational costs.
This article details how SelectDB’s data technology architect designed and implemented a new stream‑batch unified data platform using Apache Doris, covering the shortcomings of the early CDH‑based architecture, the selection process, data modeling, ingestion pipelines, performance testing, operational optimizations, and future plans.
Teradata announced its withdrawal from China due to geopolitical uncertainty and rising competition from mature domestic data‑warehouse solutions, prompting a detailed analysis of its architecture, the main Chinese warehouse designs, global market positioning, and migration tools for replacing Teradata.
Teradata's withdrawal from China, driven by geopolitical tensions and the rise of mature domestic data‑warehouse solutions, prompts a detailed look at its MPP architecture, the three main Chinese warehouse designs, Gartner market positioning, and migration tools for alternatives like GBase 8a and GaussDB DWS.
In this interview, Huawei Cloud’s chief architect Zeng Kai explains how data warehouses originated, outlines the five key fusion trends shaping their evolution, and reveals the innovative features of GaussDB(DWS) 3.0 that drive cloud‑native, real‑time, and AI‑integrated analytics.
This article details Cloud Music's year‑long data assetization journey, covering the background, practical achievements, governance methods, and future roadmap for turning massive data warehouses into high‑value, well‑governed assets that drive cost reduction and business insight.
This article presents ZhongAn Insurance's experience building real-time multi-dimensional analytics, covering application scenarios, technical challenges, the evolution of their architecture from offline to Flink‑ClickHouse and finally to StarRocks, and the principles guiding their technology selection.
This article presents a comprehensive H5 (HTML5) tracking solution that details the planning of event‑collection points, the full data‑warehouse modeling process—including schema design, retention calculations, and SQL implementations—and the automatic data‑capture mechanisms needed to improve user‑behavior analysis efficiency across the product lifecycle.
The vivo Internet Big Data team presents a standardized, extensible H5 tracking solution that automates data collection via a JavaScript SDK for navigation, focus/blur, and visibility events, incorporates privacy safeguards, and feeds a multi‑layer data‑warehouse architecture with unified ID mapping and bitmap‑based retention modeling to support comprehensive user‑behavior dashboards and future advanced analyses.
This article explores the ongoing debate between data lakes and data warehouses, clarifies their distinct purposes and technologies, discusses how they can coexist or complement each other, and introduces the concept of an integrated lakehouse architecture while promoting a comprehensive data intelligence knowledge map.
This article introduces OLAP concepts, compares major OLAP solutions such as Druid, Kylin, Doris, and ClickHouse, outlines their features and suitable scenarios, and shares practical optimization techniques—including materialized views, caching, node tiering, and query tuning—to improve performance for high‑concurrency analytical workloads.
This article presents a comprehensive overview of evolving big‑data architectures—from classic offline warehouses to Lambda and Kappa models—and details a streaming‑batch integrated solution that addresses latency, data freshness, and multi‑table join challenges to achieve minute‑level real‑time multi‑dimensional analytics.
This comprehensive guide explains the evolution from traditional data warehouses to modern data lakes, detailing concepts, architectures, differences, implementation steps, and real‑world case studies, while also comparing major cloud providers' solutions and highlighting how data platforms support digital transformation and analytics.
This article explains how Alibaba's CCO team built a scalable, real‑time experience‑insight platform using Hologres and FBI, detailing the evolution from pre‑aggregated cubes to lightweight summary tables and finally to ad‑hoc detail‑wide queries, along with practical schema, partition, and deduplication techniques.
The article details the design, core features, and real‑world performance of the DragonF MPP DB, a cloud‑native, compute‑storage‑separated database that overcomes traditional MPP limitations, supports millions of daily jobs, and outlines its future roadmap for ultra‑large‑scale data platforms.
This article introduces OLAP concepts, compares major OLAP engines such as Druid, Kylin, Doris, and ClickHouse, outlines real‑world application scenarios, and provides detailed optimization techniques—including materialized views, caching, tiered storage, and skip‑index configurations—to improve query performance.
This document outlines a comprehensive big‑data platform construction plan for a financial enterprise, describing the current data challenges, objectives, three‑layer architecture, recommended commercial Hadoop solution (TDH), detailed model‑design steps, implementation schedule, hardware/software specifications, and key success factors.
The article explains how wide tables are maintained without excessive resource consumption, outlines criteria for deciding which metrics belong in a wide table, describes their granularity and timeliness considerations, and clarifies that they do not automatically expand when tracking points change.
This article traces the history of data platforms—from the first general‑purpose computers and traditional BI, through the rise of data warehouses, big‑data frameworks like Hadoop, Spark and Flink, to the modern data‑stack era with cloud‑native architectures, Lambda/Kappa models, and emerging tools—highlighting key technologies, architectural shifts, and future prospects.
This article introduces OLAP concepts, compares ROLAP, MOLAP, and HOLAP, evaluates mainstream OLAP engines such as Druid, Kylin, Doris, and ClickHouse, and presents practical optimization techniques including materialized views, caching, tiered storage, and query tuning for large‑scale analytical workloads.
This article presents a comprehensive overview of TikTok e-commerce's near‑real‑time data lake implementation, detailing data lake characteristics, architecture choices, practical use cases across analysis and operations, and for future challenges and plans.
This article explains what data metrics are, compares offline metric testing with traditional testing, and provides a comprehensive step‑by‑step guide for testing data collection, ETL, warehouse models, metric calculations, scheduling, security, and API outputs in a Hive‑based data warehouse.
The article analyzes current data‑warehouse development trends—standardization through data governance, real‑time processing via stream‑batch integration, modular architecture, and holistic performance evaluation—while linking these trends to emerging concepts such as data middle‑platforms, data lakes, and DataOPs.
This article explains the purpose and benefits of data‑warehouse layering, outlines the four ETL steps, describes each architectural layer from ODS to ADS, presents modeling principles, naming conventions, and includes sample DDL to illustrate how layered design improves data quality, reuse, and operational efficiency.
This article traces 20 years of big‑data evolution, defines data lakes and data warehouses, compares their trade‑offs, and explains how lakehouse solutions—exemplified by Alibaba Cloud MaxCompute—merge flexibility with enterprise‑grade performance to lower total ownership cost.
This article explains the concepts, architecture, and layered design of data warehouses, covering data flow, ETL processes, ODS, DWD, DWM, DWS, ADS layers, their characteristics, differences from databases, and the role of data marts in supporting OLAP and decision‑making.
This comprehensive guide explains how metadata connects source data, warehouses, and applications, outlines its technical and business classifications, demonstrates its value for data management, profiling, portals, and ETL development, and details optimization, storage, lifecycle, and quality practices essential for robust big‑data operations.
This comprehensive guide explains the fundamentals, design rules, and various types of fact tables—including transaction, snapshot, and aggregate tables—while detailing Kimball's four-step modeling process, grain declaration, handling of additive measures, and practical examples for effective data warehouse implementation.
This comprehensive guide explains why data modeling is essential for big‑data systems, compares relational and OLAP approaches, details ER, dimensional, Data Vault and Anchor methodologies, and walks through Alibaba’s multi‑stage data‑model practice, integration framework, dimension design, fact‑table strategies and aggregation techniques.
This article reviews the practical applications and experimental explorations of StarRocks at 360, describing the cloud‑native lake‑warehouse product Yunzhou, its three‑tier architecture, performance comparisons with Trino using TPCH 100 GB, challenges of Kubernetes integration, and future directions for storage‑compute separation.
This article explains the concept of metadata, its role in data warehouses, why managing metadata is critical for building, maintaining, and scaling data warehouse systems, and outlines practical steps, use cases, and tools for effective metadata management.
This article details why 360 selected StarRocks as its OLAP engine, compares its performance and resource usage against MySQL, Hive, Spark, Druid, ClickHouse and Doris, and describes the concrete deployment scenarios and data products built on StarRocks within the company.
The article outlines comprehensive data‑engineering and warehouse‑design principles—covering collection (four Ws and methods like SDK, point‑code, binlog), reporting strategies, source selection, modeling with fact, aggregation, dimension and model tables, quality checks, and governance practices such as standardized SDKs, metric libraries, automated lineage, and cost optimization—to share actionable experience for any organization.
This case study describes how Noah Holdings migrated its self‑built CDH data platform to Alibaba Cloud, replacing Impala with Hologres to achieve faster, lower‑cost, and unified OLAP analytics that support real‑time business insights and improve overall data operations.
Facing soaring data volumes and performance bottlenecks, Noah Wealth replaced its self‑built CDH cluster with Alibaba Cloud’s unified big‑data platform and Hologres, streamlining OLAP analysis, cutting costs, accelerating queries to sub‑second response times, and enabling real‑time, multi‑dimensional analytics for its financial services.
This article explains what a data warehouse is, why modeling it matters, and compares four major modeling approaches—ER, dimensional, Data Vault, and Anchor—detailing their structures, steps, advantages, and typical use cases, while also offering guidance on selecting tools and designing models.
The article presents a comprehensive overview of Tencent Cloud's metering and billing system, detailing the billing models, multi‑dimensional data analysis, real‑time data‑warehouse construction, operator orchestration, hot‑key handling, smooth upgrade strategies, and future evolution directions for large‑scale cloud services.
This article explains how an e‑commerce company designed and implemented a modern data warehouse—combining batch Spark jobs, real‑time Flink streams, and Hudi data‑lake storage—to handle terabytes of daily logs, ensure data quality, and provide fast, reliable analytics for business decision‑making.
The article details Cloud Music’s data‑governance journey, covering early modeling standards, self‑service data tools, quality and metadata management, asset‑reuse improvements, and cost‑saving Spark optimizations, while sharing concrete metrics, processes, and the team’s systematic methodology.
The article details how 37 Mobile Games built and continuously evolved a multi-dimensional analytics platform—covering business background, data challenges, the migration from MySQL through Druid, Impala, ClickHouse to StarRocks, self‑service data tools, monitoring, and future roadmap—highlighting technical decisions and lessons learned.
StarRocks is a high‑performance MPP database whose simplified FE/BE architecture, fully vectorized engine, and CBO optimizer enable fast multi‑table joins, while its partition‑bucket‑tablet storage model supports real‑time metric services and dashboard migrations, accompanied by practical troubleshooting guidance and upcoming enhancements.
Over more than ten years, Tencent evolved its big data infrastructure through three generations—from early Hadoop-based offline processing, to a hybrid real‑time Spark/Storm system, and finally to a self‑developed, open‑source machine‑learning platform—highlighting the shift from “borrowed” solutions to fully proprietary, AI‑ready architectures.
The article explains how data lakes excel at ingesting massive, varied data, data warehouses optimize storage and query performance, and lake‑house architectures combine both strengths—offering scalable, low‑cost storage with high‑speed analytics—highlighting industry solutions from Snowflake, Databricks, and major cloud providers.
This article shares Fenbeitong’s practical experience building a big‑data platform for a young B2B company, covering company background, data‑team formation, technology selection, architecture design, governance processes, modeling tools, batch and real‑time modeling, and insights on ToB versus ToC technical choices.
This article details Xiaomi's three‑year journey of adopting Apache Doris across dozens of internal services, describing the transition from a Spark‑SQL‑based Lambda architecture to a unified MPP database, performance benchmarks, data ingestion pipelines, compaction tuning, two‑phase commit, single‑replica writes, monitoring, and community contributions.
The article describes how fragmented real‑time, batch, and online data‑warehouse pipelines suffer from low productivity and inconsistent data quality, and introduces a unified SQL engine built on Apache Calcite that parses, optimizes, and compiles a single SQL statement into executable plans for ODPS, Flink, or Java, leveraging Janino code generation, multi‑backend state storage, and snapshot‑join semantics to boost performance and simplify development.
This article outlines Snowball's data warehouse background, compares major modeling approaches such as ER, dimensional, DataVault and Anchor models, describes the current challenges of their dimensional model, and details the OneData methodology—including OneModel, OneID, and OneService—along with its practical implementation, results, and future plans.
The article compares Data Fabric and Data Mesh as modern data‑architecture approaches, explains their technical and organizational differences, discusses the ongoing debate between data lakes, warehouses, and lakehouses, and highlights how each option fits varying data‑type and usage scenarios.
This article explains Bilibili's lake‑warehouse integrated architecture, describing how Iceberg, MagnuS, Trino, and Alluxio are used to achieve flexible data storage, high‑performance query acceleration, and automated indexing through Z‑Order, Hilbert curve, Bloom filter, and advanced BitMap techniques.
Bilibili’s lakehouse architecture merges Iceberg‑based data lake flexibility with data‑warehouse efficiency, using Kafka‑Flink real‑time ingestion, Spark offline loads, Trino queries, Alluxio caching, Z‑Order/Hilbert sorting, and enhanced BloomFilter and bitmap indexes to boost query speed up to tenfold while drastically cutting file reads.
From data ingestion to real‑time analytics, this guide breaks down the essential layers of a typical big‑data platform—covering collection methods, HDFS storage, Hive/Spark analysis, data sharing mechanisms, application use‑cases, streaming with Spark Streaming, and the need for robust scheduling and monitoring.
The article presents a comprehensive end‑to‑end traffic data architecture for an e‑commerce platform, detailing hybrid frontend/backend tracking with SPM/SCM/action standards, data‑warehouse construction of fact and dimension tables, UUID i_code unification, real‑time attribution methods, and future automation of warehouse and model layers.
This article traces the evolution of enterprise data platforms—from early data warehouses to modern data lakes and the emerging lakehouse—detailing key technologies, challenges, and best practices for storage, compute engines, metadata, and integration, while highlighting how cloud-native object storage reshapes scalability and cost.
This article details Kuaishou’s real‑time data warehouse architecture and its comprehensive assurance framework—including forward lifecycle standards, reverse fault‑injection testing, and Spring Festival event practices—highlighting challenges of massive traffic, high timeliness, accuracy, and stability, and outlining future plans for automation, batch‑stream integration, and cost reduction.
This article explains how to design and implement a data‑masking service that also provides mock data generation for data‑warehouse testing, covering the architecture, pain points, masking principles, workflow, evolution into a warehouse mock service, practical scenarios, and the significant efficiency and cost benefits achieved.
This article explains the fundamental differences between transactional (OLTP) and analytical (OLAP) database workloads, describes how data warehouses isolate analytical queries, and introduces star and snowflake schema designs for efficient reporting and business intelligence.
This document presents a comprehensive overview of the traffic domain in a data warehouse, covering its concepts, objectives, guiding principles, core and extension models, data quality, monitoring, scheduling, and operational practices to achieve a complete, accurate, efficient, low‑cost, and high‑value traffic data system while addressing massive data volume, consistency, and SLA challenges.
The article proposes replacing the traditional multi‑layered data‑warehouse architecture (ODS‑DWD‑DWS‑ADS) with a single, column‑store wide‑table per business theme, achieving roughly 30 % storage savings and faster queries, while acknowledging higher ETL complexity, back‑tracking costs, and production timing challenges.
This article examines the core trade‑offs of data warehouses—freshness, query latency, and cost—compares offline and real‑time architectures, introduces the concept of a streaming data warehouse, and details how Apache Flink Table Store aims to provide a unified, low‑cost solution.
This article explores how combining Flink CDC with StarRocks can streamline real‑time data pipelines, reduce component complexity, support both full and incremental synchronization, and enable efficient OLAP queries and updates for fast, scalable analytics across diverse business scenarios.
This article explains Bilibili's lake‑warehouse integrated architecture, describing how Iceberg, Z‑Order sorting, and advanced indexing techniques such as BloomFilter and BitMap are used to accelerate queries and improve data organization in large‑scale analytics workloads.
Apache Kylin leverages pre‑computed OLAP cubes on Hadoop/Spark/Flink to deliver sub‑second query responses for massive datasets, detailing its architecture, integration with BI platforms, user security, cube building, monitoring, and storage using HBase, illustrating how it overcomes big‑data analytical challenges.
This article presents JD Retail's OLAP platform construction and practical scenarios, covering control‑plane design, architecture, business management, operational safeguards, real‑time data updates, materialized view acceleration, join optimization techniques, high‑concurrency queries, and large‑scale write throughput for e‑commerce peak periods.
The article explains the challenges of big‑data analytics, introduces pre‑computation OLAP concepts, and details how Apache Kylin together with Spark, Flink, Presto and other components can be integrated into a BI platform to achieve near‑real‑time query performance on massive datasets.
This article presents the cloud‑native redesign of ClickHouse, covering its current technical limitations in storage and computation, the proposed storage‑compute separation with DDL task management, multi‑replica and CommitLog mechanisms, and a new MPP query layer to meet future data‑warehouse demands such as real‑time analytics, flexibility, high throughput, low cost, and support for semi‑structured data.
During the 2022 China Internet Industry Application Salon, JD Cloud’s product manager explained how the Color gateway, an API gateway handling billions of daily requests, overcomes stability, high‑availability, reliability, and performance challenges during peak sales by adopting a cloud‑native ClickHouse data warehouse that boosts processing speed, reduces costs, and provides real‑time analytics.
This article details JD Logistics' journey from fragmented, massive‑scale data to a unified, real‑time analytics platform, covering business needs, pain points, tool evaluation, a new Doris‑based architecture, table management, data import procedures, automation scripts, and future roadmap for data engineering.
This article details the end‑to‑end process of migrating a 10,000‑node offline data‑warehouse from CDH 5.14.4 (HDFS 2.6.0) to HDP 3.1.4 (HDFS 3.1.1), covering version selection, rolling‑upgrade strategy, incompatibility fixes, client handling, tool coexistence, testing, automation, and lessons learned.
