This article details Lianjia's journey from a Hadoop‑based 0.0 data platform to a sophisticated 2.0 architecture, describing the three‑layer design, OLAP engine choices, transparent compression techniques, operational challenges, and practical recommendations for building and maintaining large‑scale big data systems.
This article details the evolution of Lianjia's massive‑data platform from its initial 0 version through 1.0 and 2.0, describing architectural challenges, a three‑layer redesign, data‑engine selection (ROLAP, MOLAP, Kylin), transparent compression techniques, and practical lessons for large‑scale data systems.
In this interview, Baidu’s senior big‑data architect Ma Ruyue discusses his career transition from Hadoop to online databases, the design philosophy behind Baidu’s Palo ROLAP system, the future of HTAP, and his views on the evolving big‑data ecosystem including Spark, AI, and containerization.
This article details Chain Home's OLAP platform architecture and Kylin usage, covering the evolution from early ROLAP to MOLAP multi-dimensional engine, Kylin's basic principles, platform structure, application scenarios, usage specifications, capability extensions, and middleware development.
This article introduces Hulu's in‑house OLAP engine Nesto, detailing its near‑real‑time data ingestion, nested data model, TB‑level storage using HBase and Parquet, MPP query execution, custom predicate library, and the overall architecture that enables sub‑second ad‑hoc queries for user analytics.
This article shares a data engineer's three‑year experience using Apache Kylin for real‑time OLAP on petabyte‑scale data, describing the business background, challenges of pre‑computation, cube modeling, dimension reduction, and various optimization techniques such as hierarchy, mandatory, and joint dimensions, as well as precise count‑distinct handling.
This article shares practical experience deploying MariaDB ColumnStore for offline OLAP, covering its MPP‑based architecture, user‑experience benefits, high‑availability options, numerous syntax and data‑type limitations, and detailed scripts for schema conversion and incremental data extraction using pt‑archiver.
This article presents a detailed case study of Baidu Waimai's traffic analysis platform, outlining the data challenges of high dimensionality and volume, the evaluation of OLAP engines, the adoption of Apache Kylin for pre‑computation, the end‑to‑end data modeling, cube construction, incremental builds, and integration with Saiku‑Mondrian reporting, while sharing practical lessons and performance gains.
This article walks through Rubik's OLAP cube reduction techniques—including aggregation groups, required, combined, derived, hierarchical, and partial cubes—by designing and implementing buyers and suppliers cubes with six tables, demonstrating performance gains through pre‑computed queries and SQL examples.
This article explains why raw OLAP Cubes consume exponential storage, then details six practical dimensionality‑reduction methods—Aggregation Group, Mandatory Dimension, Joint Dimension, Derived Dimension, Hierarchy Dimension, and Partial Cube—showing how each can dramatically shrink materialized tables while preserving query performance.
The article explains the Materialized View Based Optimizer (MBO) in Inceptor, detailing its purpose, configuration parameters, supported aggregation, filter, and join scenarios, and shows how combining MBO with Transwarp Rubik cubes can dramatically accelerate OLAP query performance.
Vipshop faced massive data growth that broke traditional RDBMS, causing slow OLAP queries, inefficient ETL, and long development cycles, so it iteratively rebuilt its analytics stack—adding Hadoop/Hive, a self‑service UI, Presto, Kylin, and Redis—to achieve sub‑second query responses, higher concurrency, and a flexible, low‑latency BI solution.
This article explains the fundamentals of OLAP cubes, describes how Transwarp Rubik enables visual cube design and materialization, and walks through key concepts such as dimension models, hierarchies, and the tool's management features to accelerate multidimensional analytics.
Apache Kylin is an open‑source, distributed OLAP engine built on Hadoop that uses pre‑computed cubes to deliver sub‑second, high‑concurrency SQL queries on massive datasets, integrates with popular BI tools, offers a modular architecture, recent 1.5.x enhancements, and extensive deployment options.
This article examines the convergence of OLTP and OLAP by introducing Apache Phoenix, Apache Trafodion, and Splice Machine, compares their technical features, and describes how Baidu Waimai adopted a Phoenix‑based solution to address scalability and performance challenges in its operational data store.
This article examines how Baidu Waimai designed and evolved its big data platform, comparing traditional BI to modern 4V‑driven architectures, detailing database choices, OLTP/OLAP trade‑offs, data‑analysis team structures, and the essential steps for data governance and platformization.
This article describes Hulu’s real‑world OLAP query optimization, covering the fundamentals of OLAP, comparisons of row‑ and column‑based storage formats, detailed indexing mechanisms of Parquet, ORC and CarbonData, and the specific schema, shuffle, block size, speculation and GC tuning techniques that enabled CarbonData to dramatically accelerate wide‑table queries on SparkSQL.
Alibaba's HiStore columnar database, deployed for e‑commerce historical data, processes over 6 trillion records and 5 PB of data daily, offering high compression, low cost, linear scalability, MySQL compatibility, and superior OLAP performance for massive multi‑dimensional queries.
Alibaba’s HiStore column‑store database handles over 6 trillion records and 5 PB of data daily, delivering ultra‑high compression, low hardware cost, and millisecond‑level multidimensional queries for massive e‑commerce and cloud workloads.
This article introduces Druid as an open‑source, distributed column‑store OLAP engine, explains its architecture and indexing mechanisms, discusses real‑time and batch data ingestion for order analytics at Qunar, compares it with other engines, and shares practical tips and pitfalls.
This article presents a comprehensive overview of the Qdata session on OLAP engine exploration, detailing the limitations of traditional MySQL‑based solutions, the requirements for large‑scale analytics, the architecture and theoretical foundations of Apache Kylin, its cube construction process, storage in HBase, query rewriting, real‑world flight‑ticket data applications, and the encountered challenges with corresponding optimization practices.
This article explains Druid’s purpose as a real‑time, distributed, column‑store OLAP engine, details its architecture and indexing techniques, discusses practical use cases and limitations, and shows how Caravel can complement Druid for visual analytics and detailed data access.
This article presents Qunar's experience transitioning from MySQL‑based OLAP to Apache Kylin, detailing the performance challenges, required features, Kylin's architecture and theory, cube construction process, storage mechanisms, real‑world applications, and the pitfalls and optimization practices discovered along the way.
Huawei’s CarbonData, now an Apache incubator project, is a lightweight, low‑latency columnar storage format that separates storage and compute, offering multi‑dimensional analytics, high compression, and seamless integration with Spark and Hadoop, while addressing the limitations of traditional NoSQL, search engines, and SQL‑on‑Hadoop solutions.
HiStore is a columnar database developed by Alibaba's middleware team, designed for massive OLAP workloads with high compression ratios, low storage and maintenance costs, supporting ad‑hoc multi‑dimensional queries, knowledge‑grid optimization, efficient data loading, and offering features such as approximate queries and high‑availability clustering.
HiStore, an Alibaba middleware team's columnar database, offers high‑compression, low‑cost OLAP storage for massive datasets, leveraging a knowledge‑grid engine, column‑based storage, and efficient compression to achieve multi‑dimensional ad‑hoc queries with performance gains of up to tens of times over traditional row‑based systems.
The article explains how traditional OLTP systems cannot satisfy modern big‑data analytics needs and compares OLAP, Hadoop, and MPP architectures, highlighting their data processing models, scalability, cloud‑based managed services, and practical recommendations for building effective data warehouses.
HiStore, Alibaba’s columnar database built on a patented Knowledge‑Grid, delivers ultra‑high compression (over 10:1, up to 40:1), low‑cost storage, rapid query performance, linear scalability, and seamless MySQL compatibility, making it ideal for massive OLAP workloads and real‑time analytics across diverse industries.
This article describes how the Youzan data team combined Apache Kylin, Mondrian, and Saiku into a three‑layer OLAP system, covering background, component overviews, technical architecture, schema integration challenges, count‑distinct handling, Kylin‑specific SQL quirks, and practical solutions.
This article explains how Oracle 12c's In-Memory feature enables hybrid OLTP/OLAP workloads by storing columnar data in a dedicated memory area, covering its architecture, data loading, consistency mechanisms, query acceleration techniques, and integration with RAC for high‑availability deployments.
Baidu Maps’ Data Intelligence team built a large‑scale OLAP platform using Apache Kylin, detailing the challenges of multi‑dimensional analysis on billions of rows, the architecture, custom extensions for task, resource, and monitoring management, and performance optimizations that achieve millisecond‑level SQL responses.
MariaDB 10’s multi-source replication lets a single slave pull data from multiple masters, enabling centralized querying, backup, and data consistency checks across disparate databases—ideal for OLAP, big-data aggregation, and simplifying DBA operations.