Big Data Series #6: Getting Started with Iceberg Lakehouse – Snapshots, Time Travel, and Writes

Background and Motivation

Traditional Hive external tables stored as Parquet directories suffer from costly updates, half‑written files, lack of historical replay, cumbersome schema changes, and metadata drift across engines. Iceberg defines a table as a set of snapshots, manifest files, and data files, providing versioned, consistent views of data.

Core Concepts

Catalog

The catalog resolves database.table to Iceberg metadata locations and coordinates commits. This lab uses the REST Catalog , an independent HTTP service, to give Spark, Flink, and other engines a unified entry point.

Snapshot

Each successful commit creates a new snapshot. Readers see only the current snapshot, guaranteeing atomic visibility. A failed commit leaves readers on the previous snapshot.

SELECT snapshot_id, committed_at, operation
FROM demo.db.orders.snapshots
ORDER BY committed_at;

Time Travel

Iceberg supports native time‑travel via the VERSION AS OF clause, allowing queries on any previous snapshot for debugging, audit, or rollback.

SELECT *
FROM demo.db.orders VERSION AS OF 1234567890123456789
ORDER BY order_id;

Schema Evolution

Adding columns is a first‑class operation; existing files receive NULL for the new column without rewriting data.

ALTER TABLE demo.db.orders ADD COLUMN channel STRING;

Write Conflicts

Concurrent writes use optimistic concurrency: a job commits based on the current snapshot and retries or fails if the baseline has changed. This is not a row‑level lock model.

Small‑File Issue

Iceberg guarantees table‑level consistency but does not eliminate small files; streaming writes can still produce many tiny Parquet files, requiring periodic rewrite_data_files maintenance jobs.

Architecture Diagram

The lab topology consists of HDFS (storage), Iceberg REST catalog, and Spark (compute). Runtime JARs are mounted into Spark containers.

Local Environment and Smoke Test

./scripts/up.sh 01-hadoop
./scripts/up.sh 06-iceberg
./scripts/up.sh 03-spark

After starting the services, verify the namespace with Spark‑SQL inside the Spark master container:

docker exec spark-master /opt/spark/bin/spark-sql \
  --master 'local[*]' \
  --jars /opt/spark-data/iceberg-jars/iceberg-spark-runtime-3.5_2.12-1.6.1.jar \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.demo=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.demo.type=rest \
  --conf spark.sql.catalog.demo.uri=http://iceberg-rest:8181 \
  --conf spark.sql.catalog.demo.warehouse=hdfs://namenode:8020/iceberg-warehouse \
  --conf spark.sql.catalog.demo.io-impl=org.apache.iceberg.hadoop.HadoopFileIO \
  --conf spark.hadoop.fs.defaultFS=hdfs://namenode:8020 \
  -e 'CREATE NAMESPACE IF NOT EXISTS demo.db; SHOW NAMESPACES IN demo;'

Hands‑On Steps

1. Create Table and Insert Data

CREATE NAMESPACE IF NOT EXISTS demo.db;
CREATE TABLE demo.db.orders (
  order_id STRING,
  user_id INT,
  city STRING,
  amount DOUBLE,
  dt STRING
) USING iceberg
PARTITIONED BY (dt);

INSERT INTO demo.db.orders VALUES
  ('o1001', 1, 'beijing', 99.50, '2024-01-01'),
  ('o1002', 2, 'shanghai', 120.00, '2024-01-01');

INSERT INTO demo.db.orders VALUES
  ('o1003', 1, 'beijing', 30.00, '2024-01-02');

2. Time Travel

SELECT snapshot_id, operation FROM demo.db.orders.snapshots ORDER BY committed_at;
SELECT * FROM demo.db.orders VERSION AS OF <SNAPSHOT_ID> ORDER BY order_id;

3. Schema Evolution

ALTER TABLE demo.db.orders ADD COLUMN channel STRING;
INSERT INTO demo.db.orders VALUES ('o1006', 4, 'beijing', 66.00, '2024-01-03', 'app');

4. Verify Files on HDFS

docker exec hadoop-namenode hdfs dfs -ls -R /iceberg-warehouse/db/orders | head

Expect data/ with Parquet files and metadata/ with snapshot files.

5. Optional Java Job

The IcebergOrdersJob reads a CSV, performs two append batches, prints snapshots, executes a time‑travel query, and adds a column.

cd labs/lab06-iceberg
mvn -q -DskipTests clean package

docker cp target/lab06-iceberg-1.0-SNAPSHOT.jar spark-master:/opt/spark-data/lab06.jar

docker cp data/orders.csv spark-master:/opt/spark-data/lab06-orders.csv

docker exec spark-master /opt/spark/bin/spark-submit \
  --master 'local[*]' \
  --jars /opt/spark-data/iceberg-jars/iceberg-spark-runtime-3.5_2.12-1.6.1.jar \
  --class com.bigdata.labs.iceberg.IcebergOrdersJob \
  /opt/spark-data/lab06.jar \
  /opt/spark-data/lab06-orders.csv

Inside containers use the service name iceberg-rest (not 127.0.0.1).

Job‑level .config overrides --conf settings.

When fs.defaultFS points to HDFS, absolute paths without a scheme are interpreted as HDFS; use file:// for local CSV files.

Selection Intuition: Iceberg vs Hudi

Design focus : Iceberg – open table format, snapshot‑engine decoupling; Hudi – strong incremental upsert / near‑real‑time ingestion.

Engine ecosystem : Iceberg works with Spark, Flink, Trino and many others; Hudi focuses on Spark and Flink with a complete upsert story.

Series focus : This series primarily covers Iceberg; Hudi is mentioned only for comparison.

Common Pitfalls and Troubleshooting

REST image 403 : Avoid the blocked tabulario path; use the apache/iceberg-rest-fixture image.

Iceberg jar not found : Ensure ./scripts/up.sh 06-iceberg ran and Spark has the iceberg-jars volume mounted.

Cannot connect to iceberg-rest : Verify containers share the bigdata-net network; test with curl http://iceberg-rest:8181/v1/config.

Job uses 127.0.0.1:8181 : Replace with service name http://iceberg-rest:8181 inside containers.

CSV path becomes HDFS path : Use the file:/// scheme or let the job auto‑prepend it.

HDFS warehouse path missing : Run ./scripts/up.sh 01-hadoop first; ensure /iceberg-warehouse is created.

Time travel returns unexpected data : Verify the correct snapshot_id and that the snapshot has not been expired.

Small‑file explosion : Batch writes are too granular; schedule rewrite_data_files jobs.

IDE error “Row cannot be resolved” : Compile Maven dependencies, reload project, and rebuild the jar.

Position in the Big‑Data Pipeline

Offline: Business DB / files → HDFS → Hive / Spark → (Iceberg tables stay on HDFS) → Reporting / OLAP
Realtime: Events → Kafka → Flink → write Iceberg → push to OLAP

Iceberg sits at the layer where data, after processing, is governed as a versioned table while the underlying bytes remain in HDFS.

Conclusion

Iceberg is a table format that uses snapshots to make files appear as a commit‑able, time‑travelable table.

The lab uses a REST catalog together with an existing HDFS warehouse.

Time travel lets you read historical snapshots; schema evolution lets you add columns without rewriting old files.

Small files and concurrent write conflicts still require disciplined ingestion and periodic maintenance jobs.

Code and environment are available at https://gitcode.com/qq_37953312/big-data