This article provides a comprehensive overview of data middle platforms, covering data aggregation, collection tools, offline and real‑time development, job scheduling, data governance, multi‑layer architecture, ETL processes, and various industry use cases, illustrating how enterprises build and manage unified data assets.
This article presents an in‑depth overview of Weibo's large‑scale machine learning platform, detailing its multi‑layer architecture, development workflow, CTR model evolution, and how Apache Flink is employed for real‑time data processing, sample services, multi‑stream joins, multimedia feature generation, and future roadmap plans.
Youzan's data middle platform faced cost growth outpacing business due to low utilization and storage inefficiencies; they applied utilization standards, containerization, COS storage migration, offline task optimization, and fine-grained cost-billing, achieving a 12% compute boost, 17% batch savings, 80% storage cost cut, and over 25% overall cost reduction.
iQIYI’s senior data engineer shares the evolution of its big‑data services from Hadoop to a Flink‑based real‑time computing platform, detailing architecture, monitoring improvements, StreamingSQL capabilities, business use cases like recommendation and deep‑learning data generation, and future plans for unified stream‑batch processing.
This article compiles a comprehensive set of 32 interview questions covering data warehouse fundamentals, data platform construction, modeling approaches, real‑time architectures, data quality, governance, Hive optimization, and related analytical techniques to help candidates prepare for data engineering roles.
According to Gartner, enterprises should view data middle platforms as strategic, collaborative hubs that balance data collection and connection, promote reusable analytics, integrate with digital platforms, and leverage data fabric, AI-driven tools, and graph analysis to become truly data‑driven while maintaining trust and privacy.
This article compares data warehouse, data lake, data platform, and data middle platform, explaining their definitions, architectures, strengths, limitations, and use‑case differences, and provides tables that highlight how each solution handles structured and unstructured data, governance, flexibility, and business value.
The article details Beike's Data Management Platform (DMP) built since May 2018, covering its overall architecture, data collection, processing, real-time profiling, storage solutions, application scenarios, achieved performance metrics, and future development directions.
The article details JD’s data product teams’ systematic preparation for the 618 shopping festival, covering pressure estimation, capacity expansion, stress testing, emergency downgrade strategies, dual‑data‑center isolation, high‑fidelity end‑to‑end testing, and continuous monitoring to ensure stable, real‑time data services during massive traffic spikes.
This article describes how iQIYI evaluated various OLAP engines, selected Apache Druid for real‑time analytics, detailed its architecture, identified performance bottlene‑cks in Coordinator, Overlord and indexing, applied configuration and resource‑allocation optimizations, and built a user‑friendly RAP platform to democratize real‑time data analysis.
This article presents a step‑by‑step, enterprise‑wide roadmap for constructing a cloud‑native system, covering the five dimensions of architecture, a detailed evolution diagram, three maturity stages, common pitfalls, and practical solutions ranging from service‑oriented redesign to micro‑service governance and data‑driven personalization.
This article summarizes the design, architecture, and key modules of a company-wide data lake platform—named “Tianchi”—built on HBase, Kafka, and Elasticsearch, detailing data ingestion, strategy output, metadata management, indexing, monitoring, and offline analysis, and shares lessons learned and future plans.
This article examines the design of a real‑time data platform, discussing its background concepts, modern data‑warehouse perspective, architectural layers, unified data‑collection, streaming, compute and visualization platforms, and the functional, quality, stability, cost and agility considerations required for building an end‑to‑end real‑time pipeline.
The article outlines a practical five-step methodology for constructing a data middle platform, covering data resource inventory, application planning, asset construction, detailed design and implementation, and organizational planning to guide enterprises toward effective data-driven transformation.
This article presents a comprehensive DevOps case study of Bilibili, covering its cultural background, community ecosystem, user‑centric demand management, migration to high‑performance microservices, and the implementation of logging, monitoring, and real‑time data platforms to support rapid, reliable delivery.
The case study details how a major state‑owned real estate company, with nearly 40 years of experience, built a three‑stage cloud‑native middle‑platform strategy—business, data, and technology platforms—using microservices, containers, and Tencent Cloud TSF to achieve integrated operations, data‑driven decision making, and agile infrastructure across its nationwide projects.
This article describes how Ctrip built a unified hotel data intelligence platform, evaluated various database solutions, selected ClickHouse as the primary engine, and implemented performance, high‑availability, and monitoring strategies to handle billions of records and thousands of concurrent queries.
The Data Middle Platform, described by Shi Kai, is a service‑oriented architecture that transforms raw enterprise data into reusable, real‑time APIs for business applications, bridging the gap between traditional warehouses and front‑end systems, accelerating digital transformation through unified governance, rapid development, and direct business value.
This article, originally shared by NetEase’s data expert Jiang Hongxiang on DataFun, outlines the construction of NetEase’s data foundation platform, covering database kernel insights and the implementation of the ad‑hoc query engine Impala with the distributed storage system Kudu, offering valuable big‑data engineering practices.
This article traces the origin of the mid‑platform concept, examines how major Chinese tech giants implement and classify their mid‑platforms, explains the distinction between front‑end, back‑end, and mid‑platform layers, and outlines common pitfalls and practical challenges in building and operating such platforms.
The article explains the concept, architecture, and key components of a data middle platform—including data aggregation, development, asset management, service systems, and operational and security mechanisms—while also promoting related books and a giveaway.
The article reviews the 2019 Tongcheng Elong Big Data Technology and Application Salon, summarizing six expert talks on data middle platforms, intelligent marketing, real‑time recommendation, Apache Pulsar, Chinese entity recognition, and hotel ranking models, plus event highlights and future plans.
The article describes how Tubi, North America’s largest free‑streaming service, built a model‑driven data platform using a high‑quality data warehouse, DBT‑based transformations, Kubernetes‑hosted JupyterHub, low‑latency Scala/Akka services, and automated machine‑learning pipelines to accelerate experimentation and decision‑making.
The article surveys modern big‑data architecture, contrasting Lambda and Kappa models, highlights common governance and integration pain points, and proposes an end‑to‑end platform featuring unified metadata, stream‑batch processing, one‑click ingestion, standardized modeling, intelligent query abstraction, and a comprehensive development IDE.
The article explains the data middle platform concept, its distinction from traditional big‑data platforms, the architectural principles behind Alibaba's implementation, and how real‑time ingestion, processing, and service layers enable efficient, collaborative, and scalable data-driven applications.
This article explains what a data middle platform is, why it matters, its core components—including storage, compute, IDE, workflow, API services, and data asset management—and details the layered architecture of ODS, DWD, DWT, DIM, and DWA, as well as dimensional modeling using Kimball’s methodology.
This article explains what a data middle platform is, outlines its construction process, discusses how to choose suitable products, and presents enterprise case studies, offering a comprehensive guide to building and leveraging a data middle platform for big‑data initiatives.
Drawing on NetEase Yanxuan’s experience, this article explains what a data middle platform is, why companies are building one for digital transformation and fine‑grained operations, and details its core components—including the data warehouse, data services, and BI platform—illustrated with real‑world diagrams.
He Fei explains how Youzan built a two‑layer data middle platform—combining a technology stack of offline, online and streaming components with an asset layer for cataloguing, quality, lineage and unified APIs—to tackle diverse business demands, technical complexity, and to enable cost‑optimized, reusable real‑time data services.
The article analyzes iQIYI’s big‑data middle‑platform strategy, detailing its origins, architecture, digital‑asset management, governance principles and how a unified, transparent, and compatible data platform enables user‑centric, scalable innovation across the entertainment ecosystem.
A recent data platform salon in Beijing gathered five leading experts who shared practical knowledge on data middle platforms, distributed transaction patterns, SQL audit design, Amazon Aurora's architecture, and JD's large‑scale HBase deployment, offering actionable guidance for modern enterprise data engineering.
At the 2019 DTBB expo in Bangkok, JD.com showcased its component‑driven, data‑centric technology stack that powers omni‑channel retail, detailing three key trends, a modular architecture that cuts development costs by half, and a Thailand case study that boosted store traffic and reduced delivery expenses.
The article explains Alibaba's data middle platform—its definition, methodology, organizational structure, key tools, and how it differs from traditional data warehouses and data lakes—while highlighting its role in supporting scalable, business‑centric data services and digital transformation.
The article introduces Snowball's AIBO data middle platform, detailing its storage‑compute separation architecture, core capabilities such as data integration, catalog, tagging, analysis tools, micro‑service data APIs, and outlines future enhancements for security, lineage, and continuous business‑driven iteration.
Suning’s big‑data center, built on a 30‑year retail evolution and leveraging technologies like AI, cloud, and IoT, showcases how integrated data platforms and robust security can drive smart retail, improve services for 600 million users, and create a new competitive edge.
This article reviews the evolution and key components of big data platforms at leading Chinese internet companies—Taobao, Didi, and Meituan—detailing their data sources, synchronization tools, storage layers, processing engines, and scheduling systems to provide practical guidance for building robust big data infrastructures.
Tencent’s iData analysis center selected Spark as its new computing platform because, unlike ElasticSearch, TiDB, and other MPP solutions, Spark offers iterative processing, shuffle support, robust SQL and DAG scheduling, and flexible SMP‑style data exchange, enabling efficient OLAP on billions of game‑user records.
The article details JD.com’s 618 shopping festival engineering, describing how componentized micro‑services, a unified data platform, and the Archimedes elastic scheduling system enabled billions of requests, real‑time data processing and seamless online‑offline integration without adding new server resources.
This article presents a comprehensive case study of the Dada‑JD Daojia big data platform, detailing its evolution from a MySQL‑based warehouse to a multi‑layered One Data, One Platform, One Service, Many Apps architecture, the technical challenges faced, and the strategic approaches adopted to ensure coverage, accuracy, stability, and scalability.
JD.com’s R&D team transformed its 618 and 11.11 promotion preparation from a six‑month sprint to a one‑to‑two‑month process by building a unified middle platform that combines component‑based technology and data services, enabling rapid, scalable, and cost‑effective retail operations across multiple scenarios.
From June 4‑5, 2019, the China‑hosted Big Data Industry Summit gathered more than 4,000 attendees and 60,000 online viewers to present award winners, release multiple whitepapers and standards, and hold six thematic forums and two roundtables that examined data platforms, asset management, security, law, and emerging technologies, outlining current opportunities and future challenges for big‑data growth.
Ant Group's Vice‑CTO announced the open‑source SQLFlow tool that merges SQL simplicity with machine‑learning power, aiming to lower AI adoption barriers, while chief architect He Changhua outlines a real‑time big‑data platform that fuses OLTP, OLAP, and AI for universal data intelligence.
This article presents Kuaishou’s adoption of Apache Druid for massive real‑time analytics, explains why precise deduplication is required, details the platform’s architecture, the hashset and dictionary‑plus‑Bitmap deduplication designs, concurrency handling, performance optimizations, and outlines the future roadmap, providing practical insights for big‑data engineers.
The article examines how data middle platforms transform raw, fragmented enterprise data into valuable data products through a supply‑chain approach, outlining their origins, core processes, deep‑processing techniques, and the essential capabilities needed for successful implementation.
The article explains Alibaba’s adoption of a ‘big middle platform, small front‑end’ strategy inspired by Supercell, detailing its organizational mechanism, data and product capabilities, and provides a visual overview of the architecture along with links to related in‑depth resources.
This article summarizes Wang Xinchun's 2018 DAMS China Data Asset Management Summit talk, detailing the current state, core services, responsibilities, evolution, architecture, challenges, and future directions of a large‑scale real‑time data platform built on Storm, Spark, Flink, and Kubernetes, including a unified data management approach.
The article details Ant Financial’s comprehensive data quality governance framework, covering its architecture, challenges, implementation strategies, and real‑world case studies, illustrating how the company integrates data monitoring, AI‑driven self‑healing, and rigorous release controls to ensure high‑quality data across its platform.
YouZan adopted MetaMarket’s Druid OLAP platform—featuring millisecond‑level interactive queries, high availability, horizontal scalability, and rich SQL/API query types—by configuring simple ingestion tasks that automatically manage real‑time and batch data, tiered hot/cold storage, and monitoring, while still facing ingestion limits, lack of joins, and occasional latency spikes.
This article presents NetEase Hangzhou Research Institute's experience in building a data infrastructure, covering database innovations such as InnoSQL, NTSDB, and InnoRocks, as well as the integration of big‑data components like HDFS, Spark, Impala, and Kudu to enable efficient storage, processing, and real‑time analytics.
This article details Youzan's transition from Hive to SparkSQL, covering platform architecture, usability and performance enhancements, migration strategies, automated engine selection, and future plans that together reduced resource consumption by up to 67% while handling thousands of daily jobs.
The article chronicles Alibaba's ten‑year journey of building and scaling its big data platform—from early Oracle clusters and Hadoop‑based Cloud‑Ladder 1 to the self‑developed ODPS/MaxCompute, real‑time Blink engine, and the unified DataWorks ecosystem—highlighting key technical milestones, performance breakthroughs, and operational challenges that powered successive Double 11 shopping festivals.
This article chronicles Alibaba's ten‑year journey of building and scaling its big‑data platform—from early Oracle clusters and Hadoop, through the launch of ODPS and MaxCompute, to global cloud expansion and cutting‑edge streaming innovations that now power billions of transactions each Double‑11.
Meitu's data platform, serving dozens of apps and handling billions of daily events, combines custom log collection, multi‑layer storage, offline and real‑time processing, and open data services to support personalization, anti‑fraud, analytics, and business growth.
Meitu’s data platform, serving dozens of apps with 500 million monthly active users and billions of daily events, combines the Arachnia log‑collection system, Kafka ingestion, multi‑layer storage (HDFS, MongoDB, HBase, Elasticsearch), offline Hive/MapReduce processing and real‑time Storm/Flink/Naix pipelines, supported by data‑workshop tools, staged evolution for scalability, and robust security and query‑validation mechanisms.
At Meitu’s Technology Salon, senior big‑data experts detailed the end‑to‑end architecture and stability measures of Meitu’s large‑scale data platform, introduced the high‑performance distributed bitmap solution Naix, showcased the evolution of Meizu’s user‑insight system, and highlighted Apache Kylin’s OLAP capabilities and Superset integration for scalable, real‑time analytics.
iQIYI’s unified “Tongtian Tower” big‑data platform integrates analytics, AI and open APIs to turn viewer behavior and public sentiment into market insights, personalized recommendations, smart casting and churn‑prediction tools, embedding a data‑driven culture that fuels its rapid subscriber growth and revenue surge.
The article presents a comprehensive overview of 360's Big Data Center evolution, covering business background, platform‑as‑a‑service architecture, data asset management, user‑profile unification, platform milestones, technical architecture, performance optimizations, online query capabilities, future plans, and a Q&A session.
This article shares how a fast‑growing short‑video company designed a lean big‑data architecture, introduced the ALPS foundation service, and leveraged Consul to automate CMDB, job distribution, service discovery, and monitoring, enabling efficient growth with minimal operational overhead.
This article describes Getui's end‑to‑end machine‑learning data platform, covering business use cases, the full ML workflow from data ingestion and feature engineering to model training, deployment, monitoring, and the practical tools and solutions adopted to address common challenges in large‑scale AI projects.
This article reviews Alibaba's MaxCompute evolution from ODPS to a unified, multi‑cluster big‑data platform, detailing its architecture, development tools, large‑scale deployments, performance optimizations, typical workload scenarios, and why it is the preferred choice for enterprise data processing.
This article details the design, architecture, and operational experience of Youzan's Data Platform (DP), covering its scheduling, data‑sync, service, and monitoring modules, the custom Airflow‑based task scheduler, current production metrics, supported task types, and future improvement plans.
This article shares Sogou's journey in constructing a large‑scale big data platform, covering business overview, the evolution of its operations infrastructure, productization practices, security measures, and practical tips for medium‑size teams seeking to add value from data.
This article outlines the evolution of 360’s Titan big‑data processing platform, describing the challenges of traditional script‑based development, the three architectural stages (pre‑Titan, Titan 1.0, Titan 2.0), the functional modules, the DITTO component framework, and key takeaways for building flexible, self‑service data pipelines.
Amid declining Hadoop usage reports, Suning.com’s 2018‑2020 big‑data platform case study reveals why the retailer still relies on Hadoop’s mature ecosystem, how it integrates HDFS, HBase, YARN, Hive, Spark, Flink and emerging tools, and what future resource‑management plans it envisions.
The article presents Ping An Bank's AI-driven data platform, covering business background, architectural layers, algorithmic applications such as customer segmentation, portrait, business forecasting, and graph analysis, and shares practical insights on platform design, model deployment, and the role of data product managers.
Alibaba’s senior data expert Yao Bin Hui explains how the company constructed a standardized, end-to-end big-data ecosystem—from low-level data collection and AI algorithms to data services and product platforms—enabling petabyte-scale integration and second-level response times that power both internal operations and millions of external users.
The Meituan Dianping Technology Seminar presents five expert‑led sessions covering big‑data recommendation strategies for homestays, the construction of Meitu’s data platform, personalized information‑flow implementation, cross‑city video‑conference communication systems, and the latest trends in front‑end technology, each illustrated with practical case studies.
The article analyzes how recent data‑privacy regulations force the big‑data industry to choose between building costly, comprehensive data platforms or focusing on lean, high‑value applications, and outlines sustainable profit models and strategic trade‑offs for enterprises.
This article explains how Meitu built and evolved a large‑scale data statistics platform to handle billions of users, detailing the challenges of growing data volume, the architectural shifts from simple scripts to Hadoop, and the design of modular components for job management, scheduling, execution, and future expansion.
Guardian 5.0 introduces a comprehensive security solution for the TDH big‑data platform, featuring LDAP/Kerberos authentication, unified ARBAC authorization, quota management, and multi‑layer architecture, while enhancing data protection and simplifying operations across cloud‑based deployments.
After Meituan merged with Dianping, engineers unified two massive Hadoop ecosystems across Beijing and Shanghai by breaking the project into four phases—unify, copy, switch, fuse—standardizing versions, implementing zone‑aware transfers, cross‑realm Kerberos, and federated metadata to achieve a single, reliable multi‑data‑center platform.
This article summarizes Xu Peng's DAMS 2017 presentation on selecting big‑data platform components, designing ETL pipelines, choosing analysis engines, optimizing Elasticsearch, and building a data‑driven team at Ctrip.
Alibaba's senior algorithm expert Yang Hongxia explains how the company fuses massive, heterogeneous data sources into a unified platform, builds automated tag‑production pipelines and large‑scale distributed knowledge graphs, and applies these technologies to drive smarter business decisions and AI‑enabled services.
This article examines how Baidu Waimai designed and evolved its big data platform, comparing traditional BI to modern 4V‑driven architectures, detailing database choices, OLTP/OLAP trade‑offs, data‑analysis team structures, and the essential steps for data governance and platformization.
The article summarizes Baidu Waimai’s recent “Code Talk” event, highlighting the speaker’s overview of the company’s big‑data platform evolution, its technical architecture, practical challenges such as data security and accuracy, and a lively Q&A covering storm, high availability, and metric management.
The article describes how LeKe Sports built and continuously upgraded its Hadoop‑based big data platform—from a manual ETL‑to‑Elasticsearch reporting system to a 2.0 architecture featuring Spark Streaming, SQL‑based query layers, Elasticsearch indexing, and cloud‑native storage and backup solutions—to meet rapidly growing PB‑scale data demands.
This article explains why data platforms are essential for modern SMEs, defines what a data platform is, outlines a four‑step methodology (source definition, analysis theme, ETL processing, and reporting), and shares architectural choices, team structures, common pitfalls, and practical advice for rapid, iterative implementation.
Tencent's Glacier multi‑dimensional analysis platform combines massive user‑portrait tags with routine analytical reports, delivering fast, accurate real‑time queries across countless dimensional combinations, enabling analysts and operators to perform targeted operations and insights as product data continuously evolves.
At Meitu’s third Internet Technology Salon in Xiamen on November 26 2016, over 150 senior engineers heard Meitu’s Lu Rongbin detail the company’s progression from simple rsync scripts to a scalable mobile data and open statistical platform, while Tencent’s Zhao Shiyuan showcased the Glacier multi‑dimensional analysis system for fast, tag‑driven queries, underscoring collaborative technical exchange in South China.
In this interview, Lianjia’s big‑data platform lead Lv Yi discusses his career, the motivations for joining Lianjia, the evolution of the company’s data infrastructure, architectural decisions, and the challenges of scaling a big‑data platform for real‑estate operations.
The article chronicles Taobao’s 13‑year evolution of its big data platform, detailing three phases—from a single‑node Oracle setup and the Tianwang scheduler, through a Hadoop‑based “Cloud Ladder 1” architecture with real‑time analytics, to the current MaxCompute/ODPS era with cross‑region projects and advanced data services.
This article chronicles Hadoop’s ten‑year evolution from its early Nutch roots to a mature big‑data platform, detailing its technical architecture, ecosystem growth, industry adoption, application scenarios, and future challenges in storage, resource management, compute engines, security, and analytics.
This article explains the basic principles, practical demo, platform architecture, statistical validation, sample size estimation, and reporting workflow of A/B testing used at Qunar to evaluate advertising strategies and product features, illustrating how data‑driven experiments are designed, executed, and analyzed.
This article describes Ctrip's development of a unified real-time data platform, detailing its motivations, architectural choices such as Kafka and Storm, implementation of shared schemas, resource control, monitoring, and operational lessons, as well as experiences with Storm, JStorm, and Streaming CQL.
This article describes how Meituan migrated from Hive‑SQL and MapReduce to Spark on YARN, built an interactive Zeppelin‑based development platform, created reusable ETL templates, constructed a Spark‑driven feature and data‑mining platform, and applied Spark to interactive user‑behavior analysis and large‑scale SEM services, highlighting performance gains and operational benefits.
In this interview, Xiaomi R&D architect Ouyang Chen shares the design philosophy, evolution stages, core modules, pitfalls, and future challenges of the Xiaomi advertising platform, offering practical insights for building scalable backend systems.
This article chronicles the step‑by‑step evolution of Dazhong Dianping’s data platform from 2012 to 2014, detailing changes in data models, storage and compute architecture, scheduling, monitoring, and data‑driven applications, offering practical insights for teams building early‑stage big‑data infrastructures.
This article recounts how TalkingData progressively introduced Spark into its Hadoop‑YARN based mobile big‑data platform, detailing early architectures, migration challenges, performance gains, the fully Spark‑centric redesign with Kafka and Spark Streaming, encountered pitfalls, and future plans for further optimization.
This article records a WeChat Q&A where industry experts discuss selecting open‑source big data solutions, advertising‑specific data scenarios, and share a practical lambda‑style platform architecture featuring Hadoop, Spark, Storm, Elasticsearch, Redis and MySQL.
