This article describes how 58.com introduced ClickHouse to handle massive daily user‑behavior logs, detailing its features, multi‑layer architecture, configuration management, monitoring, performance optimizations, and operational automation to build a high‑availability, low‑cost analytics platform.
This article provides an in‑depth analysis of Apache Druid V1’s column‑oriented storage format, covering its dictionary, encoded dimension values, bitmap inverted index, array handling, and how these structures are used during query execution, illustrated with diagrams and code examples.
This article examines the core data models and indexing mechanisms of three leading OLAP engines—Kylin, Druid, and ClickHouse—comparing their architectures, query performance characteristics, and suitability for aggregation versus detailed query workloads in modern big data environments.
This article details the end‑to‑end process of connecting an OLAP data source to SparkSQL and presents a comprehensive performance‑tuning guide covering cluster‑level resource allocation, single‑node On‑CPU/Off‑CPU analysis, flame‑graph profiling, Java Flight Recorder usage, and garbage‑collection optimization.
This article details JD's Golden Eye business's adoption of ClickHouse for offline and real‑time traffic data analysis, covering system architecture, data ingestion pipelines, high‑availability design, monitoring, performance optimizations, and practical trade‑offs, offering insights for large‑scale analytical database deployments.
This article explains Apache Kylin’s core principles and technical architecture, then details how major Chinese companies such as Baidu Maps, Lianjia, and Didi have deployed Kylin for large‑scale OLAP, describing their system designs, performance results, and the challenges they encountered.
Suning’s senior architect Yang Zhaohui explains how his team rebuilt the tag platform with ClickHouse, using RoaringBitmap and custom optimizations to achieve second-level queries on billions of user tags, dramatically cutting response time, reducing hardware costs, and enabling real-time marketing insights.
This article describes how 58.com designed and optimized a ClickHouse‑based OLAP platform for massive user‑behavior data, covering the reasons for choosing ClickHouse, its key features, multi‑layer architecture, configuration management, automation scripts, monitoring, performance benchmarks, and future improvement plans.
This article presents 58.com’s large‑scale big data platform, detailing its business scope, the WMDA one‑stop analytics system, the Wanxiang user‑portrait service, the technical challenges of massive daily data ingestion, multi‑dimensional analysis, OLAP engine selection (Kylin, Druid), bitmap‑based user‑group processing, scheduling, and overall data service architecture.
This article compares Apache Kylin and ClickHouse, two popular big‑data OLAP engines, by examining their technical principles, storage structures, optimization techniques, and ideal application scenarios to help readers make an informed technology selection.
This article explains how Suning built a high‑performance user‑tag platform on ClickHouse, replacing Elasticsearch with bitmap‑based storage and a new architecture that delivers sub‑second profiling queries for over 600 million users, detailing the design, implementation, and future enhancements.
The article examines the challenges big data brings to traditional technologies and surveys major OLAP solutions—MPP, batch processing, and pre‑computation—including Greenplum, Druid, Kylin, and Hadoop‑based engines, then outlines Youku’s specific use‑case selections for real‑time APIs, BI reporting, and ad‑hoc analysis.
Meituan’s sales system “Qingtian” boosted OLAP performance by migrating Apache Kylin’s build engine from MapReduce to Spark, consolidating Hive files, refining dictionary creation, applying a By‑layer algorithm, and bulk‑loading cuboid files to HBase, cutting resource consumption and halving build time, ultimately reaching a 100 % SLA.
This article details the three‑stage evolution of BEIKE's OLAP platform—from the early Hive‑to‑MySQL phase, through a Kylin‑based architecture, to a flexible multi‑engine design—explaining metric modeling, engine selection, performance trade‑offs, and future roadmap for supporting Druid, ClickHouse, Doris and real‑time analytics.
This article presents Sohu Intelligent Media's data warehouse construction practice, covering fundamental concepts, batch and real‑time processing, OLAP theory, multidimensional modeling, workflow management, data quality, metadata lineage, and security, with a focus on Apache Doris and a Lambda‑style architecture.
This article examines the data models and index structures of the three leading OLAP engines—Kylin, Druid, and ClickHouse—offering technical explanations, code examples, and a concise comparison to help readers understand their strengths, limitations, and suitable use cases.
This article explores the challenges big data brings to traditional data technologies and reviews various OLAP solutions—including MPP, batch processing, pre‑computation, and Hadoop‑based engines—while detailing Youku’s specific business scenarios and how different OLAP engines are selected to meet performance, scalability, and real‑time analysis requirements.
This article introduces ClickHouse, compares column‑store and row‑store databases, shows how migrating a 50‑million‑row MySQL table to ClickHouse reduced query time from minutes to under one second, and shares practical installation, migration, performance testing, and synchronization tips.
This article introduces ClickHouse, compares column‑ and row‑oriented storage, explains a real‑world migration from MySQL to ClickHouse that reduced a 3‑minute query to under one second, details installation, migration methods, performance results, synchronization options, and common pitfalls.
This article details JD.com's transition from a custom ad‑reporting system to Apache Doris, describing the background, challenges with the legacy platform, selection criteria, implementation of data import, pre‑aggregation, on‑site computation, and the resulting performance and operational benefits during regular operation and major sales events.
This article details how Beike's engineering team scaled Apache Kylin to handle tens of millions of daily queries by optimizing HBase configurations, reducing region count, improving data locality, addressing IO and JVM GC bottlenecks, and implementing comprehensive slow‑query detection and active‑defense mechanisms.
The article details Xiaomi's growth analytics platform evolution from a Lambda architecture using SparkSQL, Kudu, and HDFS to a streamlined MPP solution with Apache Doris, covering performance gains, real‑time data ingestion, query tuning, and operational improvements for large‑scale analytics.
This article details ZTO Express's journey of adopting Apache Kylin for OLAP, comparing it with Presto, describing platform architecture, performance gains, integration with scheduling and monitoring systems, and the practical optimizations and future plans that enabled sub‑second query responses on massive daily data volumes.
This article explains what ClickHouse is, compares column‑store and row‑store databases, shows how to migrate large MySQL tables to ClickHouse, presents performance test results, discusses data synchronization methods, highlights why ClickHouse is fast, and shares common migration pitfalls.
This article presents a comprehensive overview of NetEase's use of Impala as an OLAP query engine, detailing its architectural advantages, performance benefits, enhancements such as management servers, metadata synchronization, high‑availability via Zookeeper, expanded storage support, and real‑world deployment cases in the "Mammoth" platform and NetEase Cloud Music.
This article describes Meituan's massive OLAP workloads, the specific challenges of data scale, complex schemas, and precise counting, explains how Apache Kylin was integrated using wide tables and bitmap deduplication, compares its performance and features with Presto, Druid and other engines, and outlines future improvements.
This article explains why real‑time data warehouses are needed, outlines their core principles, compares them with offline warehouses, describes typical use cases such as real‑time OLAP, dashboards, feature generation and monitoring, and provides a step‑by‑step guide to designing, implementing, and operating a Flink‑based streaming warehouse with Kafka, HBase, and metadata management.
This article explains how ClickHouse’s column‑oriented storage, vectorized execution engine, rich DBMS capabilities, flexible table engines, and carefully designed distributed architecture enable it to handle massive workloads with sub‑second query latency, making it a standout OLAP solution.
This article explains how to evaluate business architecture, data volume, data types, and team expertise to select the most suitable database—whether relational with strong ACID guarantees for OLTP, analytical OLAP systems, or various NoSQL options—while highlighting Huawei Cloud offerings for different scenarios.
This article introduces ClickHouse as an OLAP columnar database, explains the differences between row‑ and column‑oriented storage, details a real‑world migration from MySQL using a CREATE TABLE AS SELECT approach, presents performance comparisons, discusses synchronization methods, and shares common pitfalls and solutions.
This article explains the classification of OLAP data warehouses by data volume and modeling approach, compares MOLAP, ROLAP, HOLAP and HTAP, reviews popular open‑source ROLAP systems, and details advanced performance‑boosting techniques such as MPP architectures, cost‑based optimization, vectorized execution, dynamic code generation, and runtime filtering.
This article explains why ClickHouse, the high‑performance columnar OLAP database, can return results on billions of rows within seconds, detailing its columnar storage, MergeTree engine, and how JD Cloud deploys and optimizes it on Kubernetes for scalability and reliability.
This article introduces ClickHouse, a column‑store OLAP database, explains its advantages over row‑store systems, guides installation on macOS, details migration strategies from MySQL, presents performance benchmarks showing 200‑fold speedups, and discusses synchronization methods, pitfalls, and practical tips.
This article explains the core differences between OLTP and OLAP, evaluates whether traditional OLTP databases like MySQL can handle analytical workloads, introduces benchmark queries, and provides a comprehensive overview of data‑warehouse concepts such as data sources, fact and dimension tables, multi‑dimensional modeling, and common cube operations.
Beike’s OLAP platform has progressed from a basic Hive‑MySQL batch pipeline to a Kylin‑based single‑engine solution, and now to a flexible multi‑engine architecture that uses a query‑engine layer to route metrics across Kylin, Druid, ClickHouse and Doris, dramatically cutting cube‑build times, supporting real‑time ingestion, and paving the way for further engine consolidation and automated performance routing.
This article introduces ClickHouse as a high‑performance columnar OLAP database, compares it with row‑based systems, details migration methods from MySQL, presents performance benchmarks showing dramatic speed gains, discusses data synchronization strategies, explains why ClickHouse is fast, and shares common pitfalls and solutions.
This article introduces ClickHouse as an OLAP columnar database, compares its storage model with row‑based databases, details migration from MySQL, presents performance benchmarks showing query speeds under one second for 50 million rows, and shares practical deployment tips and pitfalls.
Tencent’s game marketing analysis system “EAS” evolved from inefficient TDW HiveSQL jobs and file‑heavy real‑time pipelines to a scalable ClickHouse‑based deduplication service that processes hundreds of thousands of daily activity counts in sub‑second time, offering fast, reliable, and maintainable participant deduplication for massive marketing campaigns.
This article presents Alibaba's search‑recommendation real‑time data warehouse, describing its business background, typical use cases, key requirements, the evolution from architecture 1.0 to 2.0 with Flink and Hologres, best‑practice patterns such as row/column storage, stream‑batch integration, high‑concurrency updates, and future directions like real‑time joins and persistent dimension storage.
This article explains how Suning's big‑data center leverages ClickHouse’s columnar OLAP engine and a full‑chain monitoring platform to achieve real‑time query tracing, slow‑query analysis, cluster health checks, and resource‑level alerts across diverse business scenarios.
This article explains the fundamentals of Kylin Cube construction—including dimensions, measures, Cuboid generation, layer-by-layer and in‑memory building algorithms, storage mechanisms, and various optimization strategies such as derived dimensions, aggregation groups, row‑key design, and concurrency granularity—providing a comprehensive guide for big‑data OLAP practitioners.
This document explains the differences between OLTP and OLAP workloads, introduces the CSTORE columnar storage engine architecture, its core technologies, performance advantages, typical use cases, benchmark results, and future development plans for MySQL 8.0.
This article compares the data modeling and indexing techniques of three mainstream OLAP engines—Kylin, Druid, and ClickHouse—explaining their conversion processes, storage formats, query execution paths, and summarizing their strengths and limitations for aggregation and detail query scenarios.
This article introduces ClickHouse, an open‑source column‑oriented OLAP database, detailing its key features, appropriate use cases, installation steps, performance benchmark queries, and how it compares with other columnar storage solutions while highlighting its adoption by major internet companies.
The article details iQIYI’s migration from a Hive + MySQL OLAP stack to Apache Kylin, describing the system’s architecture, typical use cases, performance gains, independent HBase deployment, service platform for monitoring, and future plans such as automated cube building and clustering.
This article provides a detailed analysis of Apache Druid V1's column‑oriented storage format, covering dimension dictionaries, variable‑length encoded values, bitmap inverted indexes, array handling, and the physical metadata layout that enables sub‑second OLAP queries on massive datasets.
QQ Music replaced its slow Hive warehouse with a massive ClickHouse cluster, achieving sub‑second to ten‑second query latency on petabyte‑scale data, enabling real‑time analytics for non‑technical users, and following five operational best practices—ZooKeeper planning, idempotent writes, sensible partitions, read‑write separation, and localized joins—while leveraging Tencent Cloud’s managed ClickHouse service.
This article provides a comprehensive overview of Apache Kylin, covering its background, core OLAP concepts, technical architecture, installation steps, cube-building methods, real‑world enterprise deployments, and resources for further learning, illustrating how it enables sub‑second query performance on massive datasets.
This article explains how to build a systematic data metric framework—including indicator definitions, dimension management, data dictionary, and data warehouse architecture—to turn massive online‑school data into actionable business insights and support efficient decision‑making and BI visualization.
This article provides a comprehensive overview of OLAP systems, classifying them by data volume and modeling approach, comparing MOLAP, ROLAP, HOLAP and HTAP, reviewing popular open‑source products, and detailing architectural, query‑optimization, vectorization, storage and resource‑management techniques that affect analytical warehouse performance.
This article provides a comprehensive technical overview of ClickHouse, covering its complete DBMS capabilities, column‑oriented storage and compression, vectorized execution engine, relational SQL support, diverse table engines, multi‑master clustering, sharding, and the design philosophies that make it exceptionally fast for large‑scale analytical workloads.
This article provides an in‑depth analysis of Apache Druid V1's column‑oriented storage format, covering dimension structures, dictionaries, variable‑length integer encoding, inverted indexes, array handling, and how these components are used during query execution.
This article provides a comprehensive overview of OLAP data warehouses, covering classification by data volume and modeling, detailed explanations of MOLAP, ROLAP, HOLAP and HTAP, common open‑source implementations, and a deep dive into performance‑boosting techniques such as MPP architectures, cost‑based optimization, vectorized execution, dynamic code generation, storage compression, runtime filters and resource management.
Suning’s 618 shopping festival showcased a massive sales surge backed by its big‑data platform, which processed over 200 billion requests, handled 38.5 PB of daily data, and delivered 31.5 trillion computations, while Kafka and HBase sustained tens of millions of TPS to ensure a seamless consumer experience.
This article details AutoHome's use of Apache Kylin as its core OLAP engine, covering its development timeline, architecture, large‑scale cube deployment, performance optimizations, cluster upgrade experiences, and future directions such as real‑time OLAP and cloud‑native deployment.
This article details Autohome's use of Apache Kylin as its core OLAP engine, covering its architecture, large‑scale Cube deployment, real‑world business applications, a series of performance and operational optimizations, cluster upgrade experiences, and upcoming plans for real‑time OLAP and cloud‑native evolution.
QQ Music’s data team tackled massive PB‑scale, real‑time analytics challenges by migrating from Hive to a ClickHouse‑based OLAP platform integrated with Tencent Cloud EMR and Superset, achieving low‑latency, high‑availability data processing, self‑service visualization, and efficient read/write scaling for billions of daily events.
QQ Music’s new real‑time OLAP platform, built on ClickHouse, Superset and Tencent Cloud EMR, ingests petabyte‑scale streaming and batch data with SSD‑backed ZooKeeper, load‑balanced writes, optimized partitions and read/write separation, delivering second‑level query responses that are several times faster than Hive, Presto or SparkSQL and enabling self‑service BI for thousands of users.
ClickHouse is a column‑oriented, open‑source OLAP database developed by Yandex that offers high‑compression columnar storage, vectorized execution, and massive read/write throughput, making it ideal for large‑scale analytics while having specific usage scenarios and notable limitations such as lack of true transactions and secondary indexes.
This article provides a comprehensive overview of Druid, an open‑source, distributed, column‑store system for real‑time analytics, detailing its features, limitations, typical use cases, core components, external dependencies, data flow, and high‑availability mechanisms.
This article presents an in‑depth overview of Apache Flink's new OLAP engine, covering OLAP fundamentals, the three OLAP models, Flink's unified streaming‑batch‑OLAP architecture, performance optimizations, benchmark results, and future development directions.
The article chronicles Apache Kylin's evolution from an Apache incubator OLAP engine to a comprehensive analytical data warehouse, highlighting its five‑year growth, extensive enterprise adoption, core data‑warehouse features, and the community’s rebranding to better reflect its big‑data capabilities.
The article details iQIYI’s RAP platform, describing its real‑time analytics requirements, architectural evolution from RAP 1.x to 2.x, core design steps, integration of Druid, Spark, Flink, and KIS, and showcases business use cases such as membership monitoring, recommendation evaluation, and smart‑TV alerting.
The article details iQIYI's Real-Time Analysis Platform (RAP), describing its motivation, architecture evolution from RAP 1.x to 2.x, OLAP engine selection, product design workflow, integration of Druid KIS and Flink, enhanced diagnostics, and real-world applications in membership monitoring, recommendation evaluation, and smart TV alerting.
iQIYI’s Real‑Time Analysis Platform (RAP) combines Apache Druid with Spark/Flink to deliver minute‑level, low‑latency multidimensional analytics via a web wizard, supporting hundreds of streaming tasks and thousands of reports across membership, recommendation, and TV monitoring, while simplifying development and maintenance.
This article provides a thorough introduction to data warehousing, covering warehouse creation, the ETL process, OLAP/BI tools, data cube concepts, common OLAP operations, and the three main OLAP architectural models (MOLAP, ROLAP, HOLAP).
At a Tsinghua University forum, Hulu presented a comprehensive overview of its big‑data solutions for advertising and streaming, covering challenges of massive, complex data, the limits of MySQL, and advanced techniques using HBase, Protobuf, Redis batch pipelines, and its own MPP engine Nesto for high‑performance, scalable analytics.
This article outlines ClickHouse’s columnar OLAP architecture, dual‑center design, storage and write stability strategies, performance testing results, and practical query and system optimizations for handling petabyte‑scale data with high throughput and low latency requirements.
Apache Kylin is an open‑source, Hadoop‑based OLAP engine that provides sub‑second, multi‑dimensional SQL queries on massive datasets, with features such as cube pre‑computation, real‑time analytics, and seamless BI tool integration, and its latest v2.6.4 release adds numerous fixes and improvements.
This article explains how Apache Kylin, an open‑source distributed analytics engine built on Hadoop/Spark, achieves sub‑second OLAP query performance through pre‑computed cubes, a layered cuboid generation algorithm, bitmap‑based distinct counting, dimension optimization techniques, and tight integration with HBase for storage and fast SQL querying.
This article describes Meituan’s large‑scale OLAP requirements, how Apache Kylin was integrated to meet them, the architectural solutions, performance benchmarks against other engines, and future work, providing practical insights for building stable, precise, and high‑performance analytics platforms.
This article reviews the concepts, criteria, and characteristics of major open‑source OLAP engines—including Hive, HAWQ, Spark SQL, Presto, Kylin, Impala, Druid, Greenplum, and ClickHouse—providing guidance on selecting the most suitable solution for various big‑data analytics scenarios.
This article surveys the recent surge in real‑time data warehouse construction, explains why low‑latency analytics are needed, and compares the architectures and technology stacks of Alibaba, Zhihu, Meituan, and NetEase, highlighting the role of open‑source OLAP engines such as Druid and ADB.
This article provides a comprehensive introduction to data warehouses, covering their definition, architectural layers, characteristics, modeling approaches such as Inmon and Kimball, fact and dimension table design, star and snowflake schemas, and best‑practice principles for building scalable, maintainable warehouse solutions in the big‑data ecosystem.
The article explains the two main types of data processing—OLTP for real‑time transactional workloads and OLAP for analytical data‑warehousing tasks—detailing their distinct characteristics, resource demands, typical features, major vendor products, and a five‑point comparison of their design and usage.
This article explains fundamental big‑data concepts by comparing offline (batch) and real‑time (stream) computing, distinguishing real‑time queries from ad‑hoc queries, clarifying OLTP versus OLAP workloads, and outlining the differences between row‑based and column‑based storage architectures.
This article provides an in‑depth overview of Apache Kylin’s pre‑computation architecture, data‑warehouse concepts, step‑by‑step cube creation from Hive tables, and advanced optimization techniques such as derived dimensions, aggregation groups, and HBase row‑key encoding to achieve sub‑second OLAP queries on massive datasets.
This article provides an in‑depth technical comparison of Apache Kylin and Apache Doris, covering their system architectures, aggregation and detail data models, storage engines, data import processes, query execution, deduplication, metadata handling, performance, high availability, maintainability, usability, schema‑change capabilities, features, and community ecosystems.
Tencent’s iData analysis center selected Spark as its new computing platform because, unlike ElasticSearch, TiDB, and other MPP solutions, Spark offers iterative processing, shuffle support, robust SQL and DAG scheduling, and flexible SMP‑style data exchange, enabling efficient OLAP on billions of game‑user records.
Tencent’s iData Analysis Center rebuilt its backend as TGMars, replacing a rigid row‑oriented bitmap store and single‑node MapReduce pipeline with a more extensible architecture that shards user behavior bitmaps, eliminates shuffle overhead, and adds columnar storage, iterative processing and SQL‑like capabilities using Spark to overcome scalability and flexibility limitations.
This article introduces Apache Doris, a massively parallel processing (MPP) distributed database, covering its background, core architecture, key technologies such as reliability, maintenance, MySQL compatibility, materialized views, and real‑world applications at Baidu and beyond.
This article explains core data‑warehouse concepts—including DB, DW, ODS, OLTP, OLAP, BI, and the differing meanings of DM—as well as their relationships, integration examples, and why OLAP cannot replace data mining, providing a concise reference for beginners in data analytics.
This article presents Meituan's real‑world experience with Apache Druid, detailing the platform’s current usage, the usability, security and stability challenges faced, the principles and architecture of Druid SQL, the enhancements made to schema inference, logging, query safety, and the custom security extensions implemented to achieve fine‑grained access control and SSO integration.
This article explains the concepts of OLTP, OLAP and HTAP, describes the DRDS HTAP architecture—including its engine and storage layers, Fireworks Spark‑based engine, optimizer stages, and streaming capabilities—and demonstrates cross‑database MPP queries and streaming joins while outlining suitable use cases and limitations.
This article presents an in‑depth overview of Hulu’s big‑data platform, detailing its multi‑layer architecture, the design and functionality of the Sophon OLAP cache layer, and how Impala is employed for high‑performance query processing and integration with cloud‑native engines.
The article details JD's 2018 upgrade of its Business Intelligence platform, describing how unified offline OLAP with ClickHouse, Spark, and Scala, timeliness optimizations, and a React‑based visualization component library together improve data consistency, performance, and user experience for merchants.
This article introduces Kylin, a HBase‑based multidimensional analysis platform, explains its architecture and various performance optimizations—including multi‑tenant support, dimension dictionary handling, and cube size estimation—while showcasing a real‑world deployment and case study at 58.com.
HybridDB for MySQL delivers millisecond‑scale, multi‑dimensional filtering on billions of rows with hundreds of metrics by combining a high‑performance columnar engine, automatic composite indexes, and a fused MPP‑DAG pipeline, turning half‑day push preparation into seconds while supporting full SQL, spatial, and JSON data.
The article reviews the 2018 58 Group technology salon on big‑data OLAP, summarizing Xiaomi’s one‑stop OLAP architecture, 58.com’s challenges and solutions using Kylin, Druid, and UnionSQL, and the practical implementations and optimizations that illustrate modern OLAP practices.
This article examines why traditional relational databases like MySQL struggle with analytical workloads, compares ROLAP and MOLAP approaches, explains Spark’s architecture and its advantages for OLAP, and details how Alibaba Cloud’s DRDS HTAP leverages a Spark‑based engine to deliver real‑time distributed query processing.
The article reviews recent big-data trends—from Hadoop’s evolution and Spark’s in-memory advances to emerging storage like Ozone—while detailing data-warehouse models, query-optimizer techniques, and cloud-native architectures that integrate diverse data sources, enabling scalable, AI-ready analytics and modern data-lake capabilities.
The 2018 NDBC conference in Dalian gathered leading academia and industry experts to showcase cutting‑edge database technologies such as Alibaba's POLARDB, POLARDB X, SolarDB, Huawei Gauss, and AnalyticDB, while discussing cloud‑native challenges, large‑scale deployment, AI integration, and future collaborative research directions.
This article details how Beike Zhaofang leveraged the ClickHouse columnar OLAP database for large‑scale user behavior analysis, covering its architecture, key features, performance benchmarks against other engines, data ingestion pipelines, custom UDFs for funnel and retention metrics, deployment setup, and future enhancements.
This article summarizes a presentation by Ping An Insurance data engineers on Druid’s architecture, core concepts, node roles, tuning strategies, and real-world deployment for insurance analytics, illustrating how Druid enables sub‑second, high‑cardinality OLAP queries and supports both real‑time and batch processing.
This article examines the evolution and classification of distributed databases for OLAP workloads, comparing traditional RDBMS, MPP solutions such as Teradata and Greenplum, Hadoop‑based ecosystems, and newer architectures like ClickHouse and Palo, while highlighting their architectural traits, strengths, and limitations.