This article explains the concepts of data warehouses, data marts, and data lakes, illustrates why the lakehouse model emerged to bridge storage and compute, and outlines its key benefits such as flexibility, scalability, reduced redundancy, and unified analytics for modern enterprises.
This article explains what multidimensional tables are, why they outperform simple charts, and how thoughtful design—using visual encoding, statistical summaries, and interactive components—turns dense data grids into intuitive, actionable analytics tools for diverse users.
The article describes how Haodf's SRE team replaced Elasticsearch with ClickHouse to handle massive microservice logs, achieve low‑latency queries, reduce storage costs, and support real‑time monitoring, tracing, and metric analysis through columnar OLAP features, sharding, TTL, and materialized views.
This article explains master data management concepts, regulatory drivers, business benefits, key technical challenges, architectural trends such as graph databases and machine learning, and highlights leading vendors, providing a comprehensive overview for enterprises seeking modern MDM solutions.
This article explains the concept of event tracking (埋点), its importance for capturing user behavior, outlines the four‑module architecture of a tracking system, compares code‑based, visual and full tracking methods, describes data models, storage, management, and presents a practical case study with analysis techniques.
The article explains what a data lake is, its architecture and core capabilities, compares it with data warehouses, discusses its value and challenges, and reviews major open‑source platforms such as Delta Lake, Iceberg, and Hudi.
In an InfoQ interview, ByteDance’s data platform leader explains how the company operates over 15,000 ClickHouse nodes storing more than 600 PB of data, the architectural choices behind its analytics workloads, cloud‑native transformations, and the practical lessons learned from large‑scale deployments.
This article explains the two most common data‑mining frameworks—CRISP‑DM and SEMMA—detailing their six and five stages respectively, illustrating each phase with diagrams and highlighting how the iterative nature of data mining drives continuous improvement.
The article details iQIYI's comprehensive data governance initiative for event tracking (Pingback), covering definitions, timing, quality requirements, governance challenges, standardized specifications, coordinate management, testing and gray‑release processes, upgrade workflows, and data security measures that together reduced event volume by 40% and cut resource consumption in half.
The article explains how to design experiment evaluation metrics—from top‑down objectives to core, quality, and observation types—and construct an Overall Evaluation Criterion by processing, weighting, and aggregating metrics, providing a robust, scalable framework for credible AB‑test assessment and product optimization.
This article details iQIYI's comprehensive data‑governance practice for event tracking, covering the definition of pingback, the need for governance, the governance framework, coordinate management, gray‑data handling, and the upgrade process that reduced tracking volume by 40% while cutting resource consumption in half.
The article chronicles Lenovo Liancheng Zhida’s three‑stage evolution of OLAP engines—from early SQL Server scripts, through a Hadoop‑based Presto solution, to the adoption of DorisDB—detailing architecture, tool comparisons, implementation practices, and the performance and operational benefits achieved.
The article describes how Beike Real Estate consolidated multiple OLAP engines into a single DorisDB‑based platform, detailing the business challenges, DorisDB’s technical advantages, extensive performance and concurrency benchmarks, and the resulting improvements in stability, query speed, and operational simplicity across various business scenarios.
This article explores how a clear enterprise data strategy—distinguishing strategic goals from tactical steps, emphasizing clean and governed data, and integrating analytics with business missions—drives reliable outcomes and supports the experience economy through coordinated CXM platforms and data products.
This article compares ClickHouse and Elasticsearch for API log storage, detailing development activity, schema handling, query performance, statistical functions, MySQL integration, new features, and practical drawbacks, while providing concrete SQL examples and migration tips.
This article explains the growing analytical demands in the big‑data era, the limitations of traditional OLAP, and how Kyligence’s distributed OLAP engine addresses data‑island issues, multi‑dimensional and many‑to‑many analysis, unified security, and performance optimization with MDX on Spark, delivering a seamless Excel‑like experience.
This article details NetEase Cloud Music's journey of redesigning its data warehouse and governance processes to support over a billion monthly active users, covering pain points, standardization, shared services, self‑service tools, and the resulting improvements in data quality, latency, and operational efficiency.
Data warehouses store structured data centrally for reporting and analysis, while data lakes retain raw data in various formats, offering flexible, low‑cost, schema‑on‑read processing; the article explains their definitions, key differences, common misconceptions, and why many organizations now combine both to enable self‑service big‑data analytics.
The article describes a real‑time user behavior analysis platform built on a ClickHouse cluster, detailing its architecture, Hive‑to‑ClickHouse data ingestion with user‑ID routing, table designs for behavior and group data, and five analytical methods—event, funnel, path, retention, and attribution—leveraging shard‑level parallelism and custom functions for high efficiency.
This article explains the fundamental distinctions between data lakes and data warehouses, outlines five critical differences—including data retention, type support, user support, adaptability, and insight speed—and offers guidance on selecting the appropriate solution based on organizational needs and technology options.
The article compares ClickHouse, Elasticsearch, and MySQL for log storage and analysis, highlighting ClickHouse's active development, SQL support, JSON handling, performance, storage efficiency, integration options, and practical query examples to help developers decide the best backend for observability pipelines.
This article debunks eight common misconceptions about data lakes, explains why they are not mutually exclusive with data warehouses, clarifies that they are not limited to Hadoop or raw data only, and provides practical tips for building flexible, secure, and business‑driven data lake solutions.
This article reviews the evolution of Beike's OLAP platform—from the early Hive‑to‑MySQL stage, through a Kylin‑based architecture, to a flexible multi‑engine solution—detailing the design choices, metric system, engine selection criteria, encountered challenges, and future development plans.
This article examines the evolution of data lakes, outlines the challenges enterprises face with massive, heterogeneous data, and details Tencent Cloud’s native data lake architecture and its serverless Data Lake Compute service, highlighting performance, cost‑efficiency, and future development directions.
iQIYI transformed its data infrastructure from a traditional offline T+1 model to a comprehensive real‑time ecosystem—leveraging Kafka, Flink, a three‑layer Stream Data Service Platform, the Talos drag‑and‑drop pipeline, and a Druid‑based analytics platform—to enable low‑latency monitoring, personalized recommendations, ad targeting, and continuous machine‑learning workflows while planning future stream‑batch integration and lake‑warehouse convergence.
Suning’s Data Middle Platform integrates an accelerated OLAP engine, a star‑schema metric system, a visualization tool built on standardized dimensions, and a unified report portal to solve data silos, improve security, and enable enterprises to evolve into technology‑driven organizations.
Data lakes, emerging since 2020, are centralized repositories that store structured and unstructured data at any scale, offering flexible analytics, but require robust management to avoid becoming data swamps; this article explains definitions, advantages, typical architectures, and compares cloud and open‑source solutions such as AWS Lake Formation, Alibaba Cloud, Delta, Iceberg, and Hudi.
The article explains how organizations evolve from a simplistic, intuition‑based decision approach to a refined, data‑driven management cycle by introducing quantitative evaluation, the three‑stage decision model, PDCA loops, and modern data‑collection tools, while also highlighting common pitfalls that hinder effective data‑driven decision making.
This article explains the design, features, and technical implementation of Youzan’s BI platform, covering its target users, visualization workflow, supported chart types, filtering, permissions, drill‑down, calculated fields, SQL generation logic, and future development directions.
This article provides a detailed, end‑to‑end description of Laiye Technology's BI ecosystem, covering its background, development stages, data acquisition, transmission, transformation, loading, modeling, storage layers, statistical analysis, real‑time metrics, visualization, and future challenges, illustrating how the company builds a scalable, cloud‑native data‑driven platform.
The article outlines the PostgresConf.CN & PGConf.Asia 2020 conference agenda, highlighting a series of cloud‑native PostgreSQL sessions, speaker biographies, and technical topics such as data warehousing, migration, security, and performance optimization, all presented by Alibaba Cloud experts.
This article explores the challenges big data brings to traditional data technologies and reviews various OLAP solutions—including MPP, batch processing, pre‑computation, and Hadoop‑based engines—while detailing Youku’s specific business scenarios and how different OLAP engines are selected to meet performance, scalability, and real‑time analysis requirements.
This article explains MySQL 8.0 window functions, provides clear definitions, demonstrates basic and advanced usage with sample tables and SQL queries, and lists all available window functions, helping readers apply ranking and cumulative calculations in real‑world database scenarios.
This article explains the core differences between OLTP and OLAP, evaluates whether traditional OLTP databases like MySQL can handle analytical workloads, introduces benchmark queries, and provides a comprehensive overview of data‑warehouse concepts such as data sources, fact and dimension tables, multi‑dimensional modeling, and common cube operations.
This article demonstrates how to leverage ClickHouse’s parametric aggregate and higher‑order functions to perform path matching, intelligent path detection, ordered funnel conversion, retention calculation, and session statistics for user behavior analysis in a big‑data environment.
Alluxio is an open‑source, memory‑centric data orchestration layer that bridges compute frameworks such as Spark, Presto, and TensorFlow with diverse storage systems, offering high‑speed I/O, unified namespace, multi‑level caching, and easy deployment, while providing extensive documentation, download links, and community resources for rapid adoption.
This comprehensive guide explains what data analysis is, the types of data, its historical evolution, the relationship between data analysis, data science and business intelligence, essential skills, why analysis matters, and a step‑by‑step framework with models and metric design for effective decision‑making.
The article surveys the Apache big‑data ecosystem, covering Hadoop’s storage and resource management, column stores HBase and Kudu, compute engines Spark, Flink, Impala, and Presto, coordination via ZooKeeper, ingestion with Sqoop and Flume, messaging Kafka, security Ranger and Sentry, metadata Atlas, OLAP Kylin, Hive, quality tool Griffin, notebooks Zeppelin, visualizations Superset and Tableau, the TPCx‑BB benchmark, and ends with an Alibaba analysis competition notice.
This article explores the fundamentals, common schemes, pain points, and a smart end‑to‑end solution for data tracking (埋点), offering practical guidelines, architectural diagrams, and a concrete example to help engineers implement comprehensive, controllable, and efficient event collection pipelines.
This guide explains how Taro developers can instantly enable eight automatic, zero‑code data‑tracking events and custom analytics in WeChat mini‑programs by integrating Tencent YouShu via a one‑click template, CLI commands, and SDK configuration.
According to Gartner, enterprises should view data middle platforms as strategic, collaborative hubs that balance data collection and connection, promote reusable analytics, integrate with digital platforms, and leverage data fabric, AI-driven tools, and graph analysis to become truly data‑driven while maintaining trust and privacy.
This article explains how databases store data by rows or columns, compares their structures, performance trade‑offs, and use cases, and clarifies the distinction between column‑oriented systems and wide‑column stores such as BigTable and HBase.
This article provides an in‑depth, English‑language overview of data lakes, covering their definition, core characteristics, reference architectures, major cloud‑vendor implementations (AWS, Huawei, Alibaba Cloud, Azure), typical industry applications such as advertising and gaming, as well as practical guidance on building and evolving a data lake in a cloud‑native, big‑data environment.
This article explains how to build a digital ecosystem using user and product profiling, covering the concepts, motivations, construction methods, and real‑world applications to drive precise marketing, improve product experience, and achieve sustainable business growth.
This article describes how the frontend team designed and implemented a comprehensive user behavior data collection and analysis platform, covering its business value, overall architecture, SDK-based data gathering, event interception, processing pipelines, analytics dashboards, and practical insights for product and operations teams.
This article explains the purpose and types of data metrics, outlines a logical framework for analyzing e‑commerce performance from traffic to behavior to transaction, and details ten key metrics—including GMV, conversion rate, UV value, click‑through and exposure rates—along with practical interpretation tips.
This article explains the core modules of an A/B testing system, details a step‑by‑step design workflow using an internet‑finance example, and highlights key design principles such as scientific traffic allocation, sufficient data, rigorous statistical analysis, and continuous iteration for data‑driven product optimization.
The article chronicles Apache Kylin's evolution from an Apache incubator OLAP engine to a comprehensive analytical data warehouse, highlighting its five‑year growth, extensive enterprise adoption, core data‑warehouse features, and the community’s rebranding to better reflect its big‑data capabilities.
Xianyu created a highly extensible user‑behavior collection framework that standardizes data into a common ODPS schema, uses JavaScript Proxy to intercept navigation and API calls, maps business metrics via JSON, aggregates reports to cut dataset‑creation effort from days to minutes while avoiding heavy full‑tracking overhead.
This article explains the concept of granularity in data warehouse design, describing data models composed of structures, operations, and constraints, illustrating how granularity affects storage detail, query performance, and resource consumption, and recommending a dual‑granularity approach to balance efficiency and analytical depth.
The article explains what a Data Lake is, its origins, key characteristics, cost advantages, potential pitfalls such as becoming a data swamp, and compares it with traditional data warehouses, highlighting when each approach is most appropriate.
This article explains the purpose of data as a strategic asset, compares traditional databases with data warehouses, outlines key characteristics and related concepts of data warehouses, and introduces the Hadoop ecosystem components that support large‑scale data storage and analysis.
The article explains the emerging Lakehouse paradigm that combines the low‑cost storage of data lakes with the management and ACID guarantees of data warehouses, detailing its advantages over traditional architectures, core capabilities, early implementations, and its role in supporting modern AI and analytics workloads.
This article introduces the design and implementation of a web data‑tracking and analysis platform, covering its four‑module architecture, code‑based and visual tracking methods, collected event and user fields, visualization dashboards, and a practical e‑commerce search‑flow case study.
The article analyzes iQIYI’s big‑data middle‑platform strategy, detailing its origins, architecture, digital‑asset management, governance principles and how a unified, transparent, and compatible data platform enables user‑centric, scalable innovation across the entertainment ecosystem.
The article details the six‑layer big‑data architecture of an online school, chronicles its migration from Storm to Spark Streaming and finally to Flink, and showcases concrete real‑time applications such as gateway monitoring, user‑profile tagging, renewal reporting, and advertising analysis, while outlining future development directions.
This guide shows how to publish a GitHub repository directly from the master branch, share specific code lines via URL fragments, auto‑close issues with commit keywords, embed GitHub widgets, adjust language detection with .gitattributes, and view traffic and trending data, all without extra branches or tools.
The guide explains how comprehensive data collection in big‑data platforms relies on a standardized event model, passive and code‑based embedding, multi‑platform SDKs, a log‑middleware layer, precise location tracking, and an embedding management platform that supports workflow, testing, quality monitoring, and scalable infrastructure for future enhancements.
This article recounts TiDB’s evolution from a transactional MySQL‑compatible database to a hybrid‑transactional/analytical platform, detailing its architecture, performance challenges, integration with TiSpark and TiFlash, and future plans to balance TP and AP workloads.
To replace the native‑only tracking used by Xianyu after its migration to Flutter, the team built a high‑accuracy solution that mirrors the Flutter navigation stack with an index list, correctly fires enter/leave events on pushes, pops and filtered dialogs, and adds exposure detection based on 50 % visibility for 500 ms, ultimately delivering 100 % tracking accuracy in production.
Zhihu built a self‑service analytics platform on Druid, introduced a multi‑level Redis caching strategy, split long‑duration queries across multiple brokers, and added automatic cache invalidation to dramatically improve query latency and resource usage for massive daily request volumes.
This article explains the core differences between data lakes and data warehouses, their respective strengths, and how they complement each other to support both exploratory analytics and routine business reporting.
This article presents a comprehensive overview of Ant Financial's data analysis platform, detailing its departmental role, the data analysis lifecycle, the platform's evolution from version 1.0 to 3.0, core technical components such as intelligent sync and pre‑computation, and a practical case study of performance optimization.
This article surveys a wide range of data visualization tools—from basic spreadsheet solutions like Excel to advanced JavaScript libraries such as D3, GIS platforms like QGIS, and specialized charting suites—highlighting their key features, supported data sources, and typical use cases for effective big‑data analysis.
JD.com’s overseas data analysis platform, dubbed the Columbus platform, combines a lightweight data warehouse deployment with standardized, customizable BI tools to provide real‑time and offline analytics, visualization, KPI management, and future self‑service reporting and predictive capabilities for its global e‑commerce operations.
The 2018 roundup showcases ten standout data‑visualization projects—from a Spotify‑driven map of Bruce Springsteen’s songs and a real‑time solar‑lunar app, to Game of Thrones dialogue analytics, interactive world‑book maps, Apollo mission prints, child‑friendly Earth bios, mascara‑choice charts, Marvel cinematic networks, ISS construction timelines, and a poster‑ready Jupiter moons diagram.
The article examines three typical data collection problems faced by startups—unclear collection methods, chaotic tracking points, and poor collaboration between data and engineering teams—and offers practical strategies such as adopting full‑event models, appointing data architects, and securing top‑down support to achieve reliable, comprehensive analytics.
This tutorial explains why and how to use MySQL window functions, covering their concepts, syntax, various function families, practical examples such as ranking, distribution, lead/lag, first/last, and aggregations within dynamic windows, plus detailed SQL snippets and visual illustrations.
The article translates and explains a typical IoT platform architecture, outlining its core Gather‑Analyze‑Act functions, common building blocks such as device interfaces, messaging brokers, storage and analytics layers, and highlights key differentiators like multi‑tenancy, protocol support, security, and extensible rule engines.
This article curates a reading list of seven influential books covering high‑performance reading, the Pyramid Principle, comprehensive UX design, website analytics, user‑experience measurement, visualization techniques, and demand‑matching, offering practical insights for building strong product and UX skills.
This article presents eight advanced data‑visualization strategies—including conditional formatting, trend lines, rule‑based filtering, hierarchical views, sorting, formatting, comparative charts, and clear titles—to help big‑data professionals present information accurately, efficiently, and compellingly for better decision‑making.
This article walks through installing Elasticsearch 5.6.9, configuring system limits, creating indices, inserting and deleting documents, executing complex aggregation queries, and integrating Elasticsearch with Java using the TransportClient, providing a practical guide for building analytics on large‑scale data.
The article presents Meituan's Traffic Compass—a data‑warehouse‑driven traffic analysis platform for the hotel‑travel business—detailing its background, challenges, architectural layers, dimensional modeling, Kylin‑based query engine, configuration mechanisms, performance metrics, and future optimization plans.
The article highlights the rapidly expanding potential of big data, emphasizing the need for companies to embed analytics into their strategy to make faster, better decisions, while noting uneven progress across industries and emerging opportunities that widen the gap between leaders and laggards.
The article explains how to move beyond macro app metrics by collecting offline and real‑time user data, storing it in HDFS, processing it with Spark, visualizing behavior paths as state‑machine trees, and performing branch‑funnel analysis to uncover conversion bottlenecks and improve product quality.
This article provides an extensive overview of more than ninety open‑source and commercial big‑data tools—including ETL platforms, resource managers, storage systems, messaging queues, processing engines, and visualization libraries—detailing their core functions, typical use cases, and notable adopters.
The article presents a comprehensive approach to solving crash and performance issues in large‑scale mobile applications by reconstructing user interaction traces through a no‑track analytics platform, compile‑time AOP instrumentation, and unified data aggregation, ultimately improving debugging efficiency and reducing operational overhead.
This article presents an A‑to‑Z glossary of common big‑data terminology, offering concise definitions for concepts such as aggregation, algorithms, analytics, AI, cloud computing, databases, machine learning, and more, to help readers quickly grasp the core vocabulary of the big‑data ecosystem.
The article examines how extensive data collection and analysis of adult performers and their content reveal surprising demographic patterns and drive innovative business models, product development, and technology adaptations within the porn industry, illustrating the practical impact of big‑data insights beyond traditional sectors.
This article explains various enterprise log types, recommends monitoring tools like Cacti, Zabbix, Splunk, and the ELK stack, and details architectures for handling server, application, and user‑click logs using technologies such as Logstash, Elasticsearch, Kibana, Kafka, Flume, and Spark.
This article explains Ctrip's flight ticket tracking framework (UBT), detailing client‑side and server‑side event collection methods, the purpose and trade‑offs of each tracking type, metric definitions, data association challenges, common pitfalls, and best practices for reliable data‑driven analysis.
The article explains Ctrip's comprehensive User Behavior Tracking (UBT) system for flight tickets, detailing client‑side (ctm, action, trace, pv) and server‑side tracking, their implementation, data flow, key metrics, common challenges, and best practices for reliable analytics.
This article details a comprehensive e‑commerce data architecture, covering product module design, SKU handling, packaging strategies, order schema, promotion tracking, visit analytics, and reporting best practices, offering practical insights for building robust backend systems.
MariaDB Columnstore, the newly integrated InfiniDB storage engine, brings column‑store analytics to MySQL, offering P‑scale data handling, distributed deployment, and impressive performance gains over InnoDB, with detailed installation steps, kernel tuning, command‑line administration, and optimization tips for production environments.
At Meitu’s third Internet Technology Salon in Xiamen on November 26 2016, over 150 senior engineers heard Meitu’s Lu Rongbin detail the company’s progression from simple rsync scripts to a scalable mobile data and open statistical platform, while Tencent’s Zhao Shiyuan showcased the Glacier multi‑dimensional analysis system for fast, tag‑driven queries, underscoring collaborative technical exchange in South China.
The article outlines how mobile Internet merges the data‑driven society with the physical world to create a new big‑data architecture and describes the seven distinct camps—Infrastructure, Analytics, Applications, Cross‑Domain Architecture, Open‑Source, Data Sources & APIs, and Incubator & Training—that together form a comprehensive end‑to‑end big‑data solution ecosystem.
This article examines how Ctrip integrates data analysis into product design through two detailed case studies—its homestay channel evolution and the Kezutong app order‑detail redesign—highlighting tools, methodologies, and results that illustrate data‑driven decision making in product management.
This article examines why analytics requests often disappear when users leave a page, reviews traditional blocking tricks such as synchronous Ajax, busy‑wait loops, and image hacks, and then presents optimized approaches like URL or window.name transfer before recommending the modern Beacon API for reliable, non‑blocking data transmission.
An in‑depth guide outlines technology‑agnostic best‑practice techniques for building high‑performance big data analytics systems, covering data acquisition, storage, processing, visualization, and security, and explains how to address the five V’s of big data to meet demanding operational and performance requirements.
At AnDevCon, senior engineers from Twitter, Amazon, Cyanogen, Square, Eventbrite and others shared practical strategies for managing large Android teams, implementing effective testing pipelines, leveraging A/B experiments, designing for testability, handling embedded testing, contributing to open‑source, and using analytics to drive product decisions.
This article explains why sparse data in Oracle tables hampers continuous time‑series reporting, introduces the Partitioned Outer Join syntax introduced in Oracle 10g, and demonstrates step‑by‑step how to transform one‑dimensional and multi‑dimensional gaps into dense datasets using practical SQL examples.
This article presents a detailed, categorized catalogue of more than fifty open‑source big‑data projects—including Hadoop‑related utilities, analytics platforms, databases, BI solutions, data‑mining packages, query engines, programming languages, search tools, and in‑memory technologies—highlighting their primary functions, supported operating systems, and official links.
This guide explains how interaction designers can develop data‑analysis skills—from collecting and understanding data sources to applying analytical methods and communicating results—to make evidence‑based design decisions and improve product outcomes.
