An Introduction to Apache Parquet: Architecture, Data Model, File Format, and Basic Operations

Parquet is an analytical, column‑oriented storage format compatible with frameworks such as Spark, Hive, and Impala, and supports data models like Avro, Thrift, and Protocol Buffers. It has become the de‑facto standard for HDFS storage in offline data warehouses and OLAP scenarios.

Parquet Introduction – The article explains the motivation behind using Parquet, highlighting the advantages of columnar storage over row‑based storage, such as reduced I/O through column pruning and predicate push‑down, as well as more efficient compression and encoding.

Architecture Overview – Parquet is an Apache top‑level project consisting of five main modules: parquet-format , parquet-mr , parquet-cpp , parquet-rs , and parquet-compatibility . These modules define the format specifications, provide read/write implementations for Hadoop, Spark, and other ecosystems, and ensure cross‑language compatibility.

Data Model – Parquet supports a nested schema similar to Protocol Buffers. Each schema contains fields with three attributes: repetition (required, repeated, optional), type (group or primitive), and name. An example schema is shown below:

message AddressBook {
  required string owner;
  repeated string ownerPhoneNumbers;
  repeated group contacts {
    required string name;
    optional string phoneNumber;
  }
}

File Format – A Parquet file consists of file metadata, row groups, column chunks, and data pages. Row groups store rows horizontally, column chunks store column data vertically, and each data page is the smallest unit of encoding. The file also contains a magic number in the header for validation.

Basic Operations

1. Creating Parquet tables in Hive – Example DDL: create table t1 (id int) stored as parquet;

2. Converting TextFile tables to Parquet – Example commands to drop, create, and set compression (Snappy) and block size.

3. Viewing Parquet file schema – The parquet-tools utility can display schema and other metadata. Example usage:

# Run from Hadoop
hadoop jar ./parquet-tools-
.jar --help
hadoop jar ./parquet-tools-
.jar
my_parquet_file.par

# Run locally
java -jar ./parquet-tools-
.jar --help
java -jar ./parquet-tools-
.jar
my_parquet_file.par

TPC‑DS Benchmark Introduction – TPC‑DS is a standard big‑data benchmark that models star and snowflake schemas with 24 tables (7 fact tables, 17 dimension tables). The article shows how to generate a 10 GB sample dataset using dsdgen and briefly mentions performance comparison between TextFile and Parquet formats.

$ cd ~/training/tpcds/v2.3.0/tools $ nohup ./dsdgen -scale 10 -dir ~/data_10g &

Overall, the article serves as a practical guide for engineers working with Parquet in data‑warehouse environments.