Understanding Apache Iceberg: Table Format Architecture, Comparison with Hive Metastore, and Business Benefits

This second article in the Iceberg series provides a concise introduction to Apache Iceberg, an open table format designed for huge analytic datasets, and aims to give readers a first‑hand impression of its role in modern data‑lake architectures.

The author, a NetEase big‑data engineer, writes to summarize his work on Iceberg, share core implementation principles, and present internal practice cases for fellow big‑data engineers.

Iceberg is defined by its project website as an open table format for massive analytic datasets, essentially a "table format" that sits above file formats such as Parquet or ORC. A table format abstracts the logical schema, file organization, indexing, statistics, and the read/write interfaces that query engines use.

To illustrate the concept, the article first reviews file formats, using Parquet as an example. Parquet defines a data model (including nested types), a storage layout (Row Group → Column Chunk → Page), and embeds column‑level statistics and indexes in the file footer. The storage hierarchy is described as follows:

Row Group → Column Chunk → Page

Parquet files also provide Java JAR libraries that expose APIs for reading and writing, allowing engines to handle complex types, compression, and metadata automatically.

Iceberg’s table format similarly defines a schema, a partition specification (range or hash), table‑level statistics, and read/write APIs, but it adds a metadata layer that tracks files and snapshots. A diagram (omitted) shows Iceberg’s position in the data‑warehouse ecosystem, comparable to Hive Metastore but implemented as a set of JARs rather than a service.

Key differences from Hive Metastore:

Schema: Both rely on underlying file formats (Parquet/ORC) for complex types.

Partition implementation: Metastore stores partitions as directory structures, while Iceberg stores partition columns directly in the table. Example directory layout for Metastore: date=20200616/ |- hour=18/ | |- ... |- hour=19/ | |- ... Iceberg’s file‑metadata table records each file and its partition values, e.g.: +-------------------------------------------------------------------+-----------+---------------+ |file_path |file_format|partition | +-------------------------------------------------------------------+-----------+---------------+ |.../action=view/00007-...parquet |PARQUET |[442027, view] | |.../action=click/00015-...parquet |PARQUET |[442027, click]|

Statistics granularity: Metastore stores statistics at table/partition level, whereas Iceberg records statistics per file. Example Spark query showing file‑level stats: scala> spark.read.format("iceberg").load("icebergdb.action_logs.files").show(false)

Write API: Metastore adds partitions via an API; Iceberg commits a snapshot that creates a manifest, a manifests list, and a snapshot file, providing full ACID guarantees (atomicity, consistency, isolation, durability).

The commit workflow uses an optimistic‑lock pattern: a version-hint.text file stores the current snapshot version; a new version is written to a temporary file and then atomically renamed. Example HDFS command: hadoop@ntsdb2:~$ hdfs dfs -ls /libis/hive-2.3.6/hadoop_iceberg/action_logs/metadata

Business benefits of Iceberg:

Reduces NameNode list‑request pressure by eliminating directory‑level scans.

Improves query performance through partition pruning and file‑level statistics.

Enables near‑real‑time incremental consumption (minute‑level latency) via snapshot‑based reads.

Leverages open file formats (Parquet, ORC) without additional infrastructure.

Allows low‑cost schema and partition‑spec evolution via snapshot metadata.

Community roadmap (new features):

Support for Spark 3.0, adding SQL DDL/DML capabilities and performance improvements.

Hive InputFormat integration, enabling Hive SQL queries on Iceberg tables.

Flink sink/source support, already merged into the master branch.

Batch row‑level deletes for data‑compliance corrections.

In summary, the article equips readers with a foundational understanding of Iceberg’s architecture, its advantages over traditional Metastore approaches, and a glimpse of upcoming enhancements, setting the stage for the next practical implementation guide.