This article breaks down Dremel's columnar encoding for nested data, detailing the definition‑level and repetition‑level concepts, showing step‑by‑step examples of encoding and reconstructing JSON‑like schemas, and explaining the limits of single‑column reconstruction.
AliSQL integrates DuckDB as its storage engine to achieve high‑density columnar compression and fast analytical scans, detailing DuckDB’s multi‑layer storage format, adaptive compression algorithm selection, performance benchmarks versus InnoDB, HBase, ClickHouse, OceanBase, and the engineering optimizations AliSQL adds for throughput and cost reduction.
This article explains the challenges of frequent column changes in AI feature engineering, introduces Paimon’s column‑separation storage with a global continuous Row ID, details its Blob data type for efficient multi‑modal handling, and outlines production results and future roadmap for building an AI‑native data lakehouse.
The article explains how Apache Parquet’s columnar layout, multi‑level row‑group structure, projection and predicate push‑down, and advanced compression and encoding make it the high‑performance, space‑efficient storage format that powers modern big‑data ecosystems and tools like Spark, Python pandas, and ClickHouse.
StarRocks 4.0 introduces FlatJSON, a columnar storage and execution engine that converts high‑frequency JSON fields into native columns, dramatically reducing I/O and CPU costs and enabling JSON queries to run with performance close to that of traditional columnar data.
This article examines how adaptive structural encoding in columnar formats like Lance dramatically improves random‑access performance on NVMe storage, compares it with Apache Parquet and Arrow, and discusses the trade‑offs between scan speed, memory usage, and compression.
This article details OceanBase's evolution from a distributed OLTP system to a unified HTAP database, covering its cost‑based optimizer, vectorized execution, integrated row‑column storage, bypass import, materialized views, external tables, full‑text search, and real‑world use cases for real‑time analytics.
This article explores ClickHouse’s high‑performance OLAP architecture, covering its MPP design, columnar storage, vectorized execution, pre‑sorting, table engines, data types, sharding and replication strategies, as well as index designs that together enable rapid analysis of massive datasets.
This article explores ClickHouse’s high‑performance OLAP design—including its MPP architecture, columnar storage, vectorized execution, pre‑sorting, sharding, replication, index strategies, and compute engine—showing how each innovation contributes to ultra‑fast, scalable data analysis in the big‑data era.
This article explains ClickHouse’s core architecture—including its storage‑compute integration, MPP parallelism, columnar storage, vectorized execution, data pre‑sorting, table engines, sparse and auxiliary indexes, and the two‑stage aggregation pipeline—then walks through the exact GROUP BY execution flow for both local and distributed tables, illustrating each step with diagrams, SQL demos, and code snippets.
This article examines ClickHouse’s high‑performance OLAP design, covering its MPP architecture, columnar storage, vectorized execution, pre‑sorting, table engines, extensive data‑type system, sharding and replication strategies, as well as its sparse and skip‑index mechanisms that together enable ultra‑fast analytics on massive datasets.
ClickHouse is an open‑source, massively parallel, column‑oriented OLAP database that integrates its own columnar storage, vectorized batch processing, pre‑sorted data, diverse table engines, extensive data types, sharding with replication, sparse primary‑key and skip indexes, and a multithreaded query engine, delivering high‑throughput real‑time analytics on massive datasets.
This article explains how ClickHouse’s columnar storage, bitmap functions, and distributed architecture can be leveraged to store billions of tag bitmaps, combine them efficiently, and support fast group calculations for customer data platforms, while addressing data‑warehouse integration, storage format, and performance challenges.
Tencent Cloud's TDSQL shattered world records in both TPC‑DS (OLAP) and TPC‑C (OLTP) benchmarks, achieving a 7260 M QphDS score at a cost of 37.52 CNY/kQphDS, and the article explains the three self‑developed technologies—MPP execution, parallel execution framework, and columnar‑vectorized engine—that made this performance possible.
This article compares ClickHouse and MySQL by examining benchmark results, MPP architecture, columnar storage, compression techniques, vectorized execution, and index designs, showing why ClickHouse delivers dramatically higher query performance on massive data sets.
TeleDB’s journey from a centralized MySQL/PostgreSQL‑based system to a native distributed HTAP database showcases innovations such as share‑nothing architecture, columnar storage, vectorized execution, Remote Data Access, global caching, and advanced dead‑lock detection, dramatically improving query performance, storage efficiency, and scalability.
This article surveys two decades of database system architecture innovations—including columnar stores, cloud databases, data lakes, NewSQL, hardware accelerators, and blockchain databases—highlighting their motivations, trade‑offs, and the shifting landscape that shapes modern DBMS design.
Vivo's unified content platform replaced its unwieldy 60 TB MongoDB store with HBase, then upgraded the cluster, introduced table‑specific connection pools, column‑only reads, tuned compaction, and leveraged multi‑version cells, cutting response times from seconds to under ten milliseconds and dramatically lowering operational costs while boosting read/write performance.
This article explains why new data systems are needed, introduces Apache Arrow and its columnar in‑memory format, describes dynamic read‑time modeling, outlines the system’s execution flow, storage and indexing strategies, and shares practical tips and extensions for building scalable big‑data solutions.
This article explains why new data systems are needed, introduces Apache Arrow’s columnar in‑memory format and its zero‑copy advantages, describes how to model data at read time, outlines the execution flow with Acero and SQL planning, and shares practical tips and extensions for building robust, dynamic‑schema data platforms.
The article explains how Apache Arrow’s columnar, cross‑language in‑memory format enables high‑performance, interoperable data systems—replacing traditional row‑oriented databases—by supporting dynamic schemas, zero‑copy data exchange, efficient indexing, Acero‑based query execution, and Flight/ADBC connectivity, while offering practical guidance and highlighting challenges.
This article explains why new data systems are needed, introduces Apache Arrow and its columnar in‑memory format, describes read‑time modeling and dynamic schema handling, and shows how Arrow can be used to build a complete data processing pipeline with indexing, SQL planning, and zero‑copy data exchange.
This article explains the nature of semi‑structured data, compares traditional schema‑on‑read and schema‑on‑write approaches, and details MaxCompute's columnar storage solution that balances flexibility, performance, and cost for large‑scale data warehouses.
The interview with Wu Li, a research engineer at Shanghai Yanhuang Data, explores how columnar storage, JIT compilation, and push-mode processing are reshaping modern database performance, highlighting hardware constraints, software optimizations, and product‑centric goals in the era of big data analytics.
In this technical interview, Wu Li, a research engineer at Shanghai Yanhuang Data, explains how hardware constraints drive database evolution, why columnar storage and SIMD acceleration are crucial for OLAP, and how JIT compilation and push‑mode processing improve query performance and product experience.
The article presents an interview with Wu Li, a senior R&D engineer at Shanghai Yanhuang Data, discussing how columnar storage, JIT compilation, and push‑mode execution are reshaping database performance in the era of big‑data analytics and evolving hardware constraints.
The article presents an interview with Wu Li, a senior engineer at Shanghai Yanhuang Data, discussing how columnar storage, JIT compilation, and push‑mode processing are reshaping database performance and product strategy in the era of large‑scale data analytics.
This article explains why time‑series databases are ideal for massive telemetry data, describes the high‑cardinality problem that degrades performance, and shows how openGemini’s newly introduced columnar engine—combined with sorting and clustering indexes—effectively mitigates those issues while delivering impressive write and query speeds.
This article explains the nature of semi‑structured data, compares schema‑on‑read and schema‑on‑write approaches, and shows how Alibaba Cloud MaxCompute leverages columnar storage and dynamic parsing to achieve low‑cost, high‑performance analytics for large‑scale data workloads.
This article explains how a data‑ecosystem team governs and applies behavior‑analysis data by describing common analysis scenarios, data‑warehouse modeling methods and their pros and cons, the concepts and overall architecture of behavior‑centric analytics, key system components, and several concrete analysis examples such as retention, funnel and path analysis.
This article explains the motivation, design, and implementation of Alibaba Cloud's SLS Schema‑on‑Read scanning mode, showing how it enables SQL analysis on raw log data without pre‑built indexes, improves flexibility for evolving schemas, and reduces storage and index costs in various log‑analysis scenarios.
This article explains how ClickHouse achieves high query performance by leveraging storage‑engine designs such as pre‑sorting, columnar layout, and block‑level compression, and by exploiting a vectorized compute engine while avoiding joins and using built‑in functions.
This article explains the differences between row-based and column-based storage, comparing their write and read performance, outlining advantages and disadvantages, and describing suitable scenarios such as OLAP queries, column families, compression, and indexing, to help choose the appropriate storage model.
The article presents a comprehensive technical overview of MercsDB—formerly HermesDB—including its background, storage and indexing designs, native and Presto computation engines, vectorization optimizations, benchmark results, real‑world applications, and future development plans.
A comprehensive TPC‑H benchmark compares ClickHouse, Oracle, and the open‑source esProc SPL across simple and complex queries, showing ClickHouse excels at single‑table scans, while SPL consistently outperforms both in complex calculations and offers more concise code.
This article benchmarks ClickHouse, Oracle, and the open‑source esProc SPL on the TPC‑H suite, showing ClickHouse excels at simple scans, Oracle handles many complex queries, while SPL consistently outperforms both in speed and code simplicity across a range of workloads.
The article reports the June 2022 China database popularity ranking, highlights TiDB's comeback, introduces OtterTune's new financing, announces PostgreSQL 15 Beta 1, explains Google AlloyDB columnar features, and provides a detailed overview of columnar database concepts, history, advantages, and evolution.
This guide explains ClickHouse fundamentals, OLAP versus OLTP concepts, columnar storage benefits, core performance techniques, the MergeTree family and its indexing, specialized table engines, installation on Linux, Docker deployment, and integration with HDFS, MySQL, and Kafka for modern analytical workloads.
UniqueMergeTree, a new ClickHouse table engine, addresses real‑time data update challenges by combining upsert semantics, unique key enforcement, and efficient delete‑bitmap handling, offering higher query performance at modest write cost, with detailed design, sharding strategies, conflict resolution, and performance evaluation.
The talk outlines IoT data classifications, requirements, and proposes a one‑stop storage product using multi‑model support, columnar formats, compute‑storage separation, tiered storage, and query optimization to achieve ten‑fold cost reduction and ten‑fold performance gains.
After months of using MongoDB for frontend monitoring logs, the data grew to billions of records causing severe query slowdown, prompting a migration to ClickHouse where columnar storage, partitioning, and OLAP capabilities dramatically improved performance and storage efficiency.
This article explains how Apache Doris adopts CPU‑level vectorization and columnar storage to boost query performance, details the design and current status of its vectorized engine, and outlines future work such as JOIN acceleration, storage‑layer vectorization, import optimization, and extensive SQL function support.
This article recounts the author's journey from using MongoDB for front‑end monitoring logs to migrating to ClickHouse, detailing the challenges with large‑scale data, optimization attempts, the fundamental differences between row‑ and column‑oriented databases, and the resulting performance and storage improvements.
This article explains how Apache Doris adopts CPU vectorization techniques—such as SIMD, columnar storage, and cache‑friendly designs—to boost query performance, detailing its current vectorized engine architecture, recent benchmarks, ongoing work on JOIN, storage, import, and future enhancements.
This article provides a comprehensive overview of Tencent's self‑developed MySQL kernel TXSQL, covering its evolution, overall architecture, columnar storage engine, instant DDL capabilities, enterprise‑grade features, high‑availability mechanisms, performance optimizations, and the rigorous development and testing processes behind the product.
This article provides a comprehensive introduction to Apache Parquet, covering its origins, columnar storage advantages, nested schema support, internal architecture, storage model components, comparison with ORC, and practical tools for inspecting Parquet files.
Kudu is an open‑source columnar storage engine from Cloudera that combines high‑throughput batch processing with low‑latency random reads, offering features such as C++/Java APIs, Raft‑based replication, flexible consistency, partitioning, and integration with Hadoop, Spark, Impala, and other ecosystem components.
Tongcheng Yilong’s BaikalDB deployment combines high‑availability multi‑Raft HA, HTAP support, and share‑nothing scalability to deliver over 72K TPS OLTP and ten‑fold faster OLAP queries while cutting operational costs up to a hundredfold through dual‑center, columnar storage and cloud‑native elasticity.
ClickHouse is an MPP column‑oriented DBMS that combines full DBMS functionality, advanced columnar storage with high compression, SIMD‑based vectorized execution, a rich relational SQL interface, diverse table engines, multi‑master clustering, and flexible sharding and distributed query capabilities, making it exceptionally fast for analytical workloads.
ClickHouse is a high‑performance MPP column‑store DBMS that combines complete DBMS functions, column‑oriented storage with aggressive compression, SIMD‑based vectorized execution, flexible table engines, multithreading, distributed processing, a multi‑master architecture, and SQL compatibility to deliver fast online analytical queries on massive data sets.
This article details how the DTCC2020 guest speaker from Tongcheng‑Elong introduced BaikalDB’s distributed columnar storage, covering internal and external motivations, technology comparison, architecture, implementation tricks, performance gains in production, and future hybrid row‑column research directions.
This article provides a comprehensive overview of Apache Parquet, covering its modular architecture, nested data model, striping/assembly and definition level algorithms, file format details, push‑down optimizations, performance benchmarks, and the project's evolution within the big‑data ecosystem.
This article explains the ORC (Optimized Row Columnar) file format, its advantages, internal structure, data model, compression mechanisms, and how to create Hive tables and write ORC files using Java, providing practical code examples and reference resources.
This article explains what ClickHouse is, compares column‑store and row‑store databases, shows how to migrate large MySQL tables to ClickHouse, presents performance test results, discusses data synchronization methods, highlights why ClickHouse is fast, and shares common migration pitfalls.
ClickHouse is a PB‑scale, open‑source columnar OLAP database that uses a ZooKeeper‑coordinated sharded cluster, columnar storage, vectorized execution, advanced compression, data‑skipping indexes, and materialized views to deliver high‑performance interactive analytics, yet it requires manual shard management, lacks a mature MPP optimizer, and handles real‑time single‑row writes poorly.
This article systematically examines the core technologies of OLAP systems, covering storage models, columnar formats, indexing, distributed storage architectures, query execution steps, optimizer designs, and emerging trends such as real‑time analytics, HTAP, cloud‑native deployment, and hardware acceleration.
This article explains how JD Cloud’s JCHDB, built on ClickHouse, achieves millisecond‑level queries on billions of rows through columnar storage, distributed multi‑master architecture, SIMD vector engine, sparse indexing, and specialized table engines, and outlines the ideal use cases and deployment details.
The article explains SAP HANA’s performance advantages by combining four key technologies—high‑speed memory, columnar storage, data compression, and an insert‑only model—detailing their individual pros and cons, how they complement each other, and the trade‑offs involved in scaling and persistence.
This document explains the differences between OLTP and OLAP workloads, introduces the CSTORE columnar storage engine architecture, its core technologies, performance advantages, typical use cases, benchmark results, and future development plans for MySQL 8.0.
Apache Parquet is a language‑agnostic, columnar storage format for the Hadoop ecosystem that offers high compression, efficient I/O through column and predicate push‑down, nested‑structure support, and a three‑layer architecture, and is compared with ORC while providing tooling for schema inspection.
This article explains how Alibaba's MaxCompute platform evolved its storage engine from row‑based CFile to the columnar AliORC format, details the technical innovations such as async prefetch, small I/O elimination, adaptive dictionary encoding, and range‑aligned reads, and compares its performance against Apache ORC and Parquet.
This article provides a comprehensive overview of Apache Druid's distributed column‑store architecture, detailing its node types, external dependencies, data flow, and operational mechanisms, and compares Druid's real‑time analytics capabilities with systems such as Impala, Elasticsearch, and Spark.
This article presents a comprehensive overview of Apache Doris, covering its positioning as a distributed MPP analytical database, core architecture with FE and BE nodes, key technologies such as vectorized execution and materialized views, integration with Kafka and Elasticsearch, additional features, roadmap, and detailed case studies from Baidu Statistics and Meituan, illustrating its practical deployment and performance characteristics.
This article provides a comprehensive overview of Apache Parquet, covering its purpose, architectural components, nested data model, file structure, practical Hive commands for creating and inspecting Parquet tables, and a brief introduction to the TPC‑DS benchmark for performance testing.
This article explains the differences between row‑oriented and column‑oriented storage, compares their trade‑offs, and introduces HBase’s column‑family architecture, including row keys, column qualifiers, timestamps, cells, and how it maps to a multi‑dimensional map structure.
The article details a real‑world MySQL performance case where a sudden 100‑fold load increase was mitigated through read/write splitting, replica‑based statistics, limited index tuning, middleware‑driven sharding, and finally a columnar storage layer (Infobright) with scripted dynamic data synchronization, achieving dramatic latency reductions and scalable architecture.
A MySQL‑based receipt‑tracking service suffered a sudden 100‑fold load increase, prompting a step‑by‑step optimization that combined read‑write splitting, middleware‑less data routing, columnar storage with Infobright, and dynamic scheduling to dramatically lower CPU/IO pressure and restore performance.
HybridDB for MySQL delivers millisecond‑scale, multi‑dimensional filtering on billions of rows with hundreds of metrics by combining a high‑performance columnar engine, automatic composite indexes, and a fused MPP‑DAG pipeline, turning half‑day push preparation into seconds while supporting full SQL, spatial, and JSON data.
This article introduces Hulu's in‑house OLAP engine Nesto, detailing its near‑real‑time data ingestion, nested data model, TB‑level storage using HBase and Parquet, MPP query execution, custom predicate library, and the overall architecture that enables sub‑second ad‑hoc queries for user analytics.
Apache CarbonData 1.0.0, now an Apache incubating project, adds over 80 new features and bug fixes—including a new data loading solution, Spark 2.1 integration, update/delete SQL support, adaptive compression for numeric types, B‑Tree LRU cache, V2 format for faster first‑query performance, vectorized reader, bucket‑table joins, off‑heap memory, single‑pass loading, and pre‑generated dictionaries—aimed at delivering faster, more flexible, and efficient columnar storage for big‑data workloads.
This article describes Hulu’s real‑world OLAP query optimization, covering the fundamentals of OLAP, comparisons of row‑ and column‑based storage formats, detailed indexing mechanisms of Parquet, ORC and CarbonData, and the specific schema, shuffle, block size, speculation and GC tuning techniques that enabled CarbonData to dramatically accelerate wide‑table queries on SparkSQL.
This article explains Dameng's columnar storage architecture, the smart index mechanism that leverages zone statistics to reduce I/O, and the adaptive compression algorithms—including dictionary, constant, RLE, and sequence encoding—used to achieve high compression ratios on columnar data.
This article explains IBM DB2 BLU Accelerator’s columnar storage, multi‑level compression, TSN‑based logical rows, SIMD processing, intra‑parallel execution, probability‑based caching, and automatic admin features, showing how these technologies together deliver dramatic I/O and performance gains for analytical workloads.
This article explains four fundamental database data models—key‑value pair, document, column, and graph—detailing their structures, scalability characteristics, and typical implementations such as Redis, MongoDB, HBase, and Neo4j.
This article provides a comprehensive analysis of SQL‑on‑Hadoop architectures, comparing runtime‑framework‑based engines like Hive with MPP‑style engines such as Impala, detailing core components, compilation pipelines, optimizer strategies, CPU/IO performance tricks, columnar storage formats, and resource management in modern big‑data query platforms.
