Building Real-Time Data Lakes with Spring Boot, Flink CDC 3.0 & Iceberg: Production Patterns

1. Why Replace the Legacy Architecture?

The previous Lambda architecture (Canal + Kafka + Flink + Hive) caused operational pain: too many components, long pipelines, and frequent failures when upstream schemas changed — requiring midnight fixes.

Flink CDC addresses these with unified full/incremental sync and native schema evolution, eliminating batch-then-stream cutover and DDL-induced crashes. For Java developers, fewer middleware components mean simpler maintenance.

2. Spring Boot + Flink: The Correct Pattern

2.1 Dependency Management

Align versions carefully: Flink 1.18.0, Flink CDC 3.0.0, Spring Boot 2.7.18, Java 11. Key dependencies:

<properties>
  <java.version>11</java.version>
  <spring.boot.version>2.7.18</spring.boot.version>
  <flink.version>1.18.0</flink.version>
  <flink.cdc.version>3.0.0</flink.cdc.version>
</properties>

<dependencies>
  <!-- Flink Core & CDC -->
  <dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-streaming-java</artifactId>
    <version>${flink.version}</version>
  </dependency>
  <dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-mysql-cdc</artifactId>
    <version>${flink.cdc.version}</version>
  </dependency>

  <!-- Iceberg Sink -->
  <dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-iceberg</artifactId>
    <version>${flink.version}</version>
  </dependency>
</dependencies>

2.2 Configuration Encapsulation

Spring Boot acts as orchestration/config client — never run env.execute() in main(). Jobs are packaged and submitted to YARN/K8s. Flink environment configured via application.yml and @Configuration:

@Configuration
public class FlinkEnvConfig {
  @Bean
  public StreamExecutionEnvironment flinkEnv(
      @Value("${flink.parallelism:4}") int parallelism,
      @Value("${flink.checkpoint.interval:60000}") long cpInterval) {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    env.setParallelism(parallelism);
    // Enable Exactly-Once checkpointing
    env.enableCheckpointing(cpInterval, CheckpointingMode.EXACTLY_ONCE);
    // Retain checkpoints on cancellation for recovery
    env.getCheckpointConfig().setExternalizedCheckpointCleanup(
        CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
    return env;
  }
}

3. Full/Incremental Unified Sync: Chunk Mechanism

Flink CDC doesn't simply SELECT * then read binlog — that would OOM on large tables. Instead it uses Chunk splitting : partition by primary key into small chunks, each read as an independent mini-transaction with position tracking. After all chunks finish, seamlessly switch to binlog listening. Failed tasks resume from the last incomplete chunk.

MySqlSource<String> mysqlSource = MySqlSource.<String>builder()
  .hostname("127.0.0.1").port(3306)
  .databaseList("biz_db").tableList("biz_db.orders")
  .username("cdc_user").password("xxx")
  .scanStartupMode(StartupMode.INITIAL) // default unified sync
  .chunkMetaGroupSize(1024) // control chunk size to avoid memory pressure
  .deserializer(new JsonDebeziumDeserializationSchema())
  .build();

env.fromSource(mysqlSource, WatermarkStrategy.noWatermarks(), "MySQL CDC Source");

4. Streaming ETL & Dual-Stream Join Pitfalls

Use DataStream API for simple filters/mappings (better performance); switch to Flink SQL for multi-table joins and complex aggregations.

4.1 Dual-Stream Join in Practice

Wide-table construction is the hardest part. Flink SQL's Interval Join controls state size, preventing unbounded growth. Streams must be registered as views with time attributes:

StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);

// Register with proctime for Interval Join
tEnv.createTemporaryView("orders", orderStream,
  $("id"), $("amount"), $("user_id"), $("proc_time").proctime());
tEnv.createTemporaryView("users", userStream,
  $("id"), $("name"), $("level"), $("proc_time").proctime());

// Join orders and users within 1-hour window
String joinSql = "SELECT o.id, o.amount, u.name, u.level "
  + "FROM orders o "
  + "JOIN users u ON o.user_id = u.id "
  + "AND o.proc_time BETWEEN u.proc_time - INTERVAL '1' HOUR "
  + "AND u.proc_time + INTERVAL '1' HOUR";

Table resultTable = tEnv.sqlQuery(joinSql);

5. Schema Evolution: No More Midnight Code Changes

Previously, upstream schema changes required downstream code + Iceberg table modifications + job restarts. CDC 3.0 automates this by splitting the stream into DataEvent and SchemaChangeEvent. Iceberg Sink with schema evolution enabled auto-applies DDL changes.

For whole-database sync, CDC 3.0 recommends YAML configuration over Java code:

# pipeline.yaml
source:
  type: mysql
  hostname: localhost
  port: 3306
  username: root
  password: password
  tables: db\..*

sink:
  type: iceberg
  catalog-name: my_catalog
  catalog-properties:
    type: rest
    uri: http://localhost:8181

pipeline:
  name: mysql-to-iceberg
  parallelism: 4
  schema.change.behavior: evolve  # enable auto schema evolution

In Spring Boot, parse this YAML and submit via PipelineBuilder. For incompatible changes (e.g., column drops), configure try_evolve or exception based on business tolerance.

6. Iceberg: Goodbye Painful Hive Partitioning

Hive required explicit partition keys like dt=2023-10-01; missing them in queries triggered full scans. Iceberg's hidden partitioning is elegant:

-- Create table with hourly hidden partition
CREATE TABLE orders (
  id BIGINT,
  amount DECIMAL(10, 2),
  create_time TIMESTAMP(3)
) PARTITIONED BY (hours(create_time));

Users query WHERE create_time > '2023-10-01 12:00:00' — Iceberg automatically prunes partitions without user awareness.

Writing to Iceberg:

Table icebergTable = catalog.loadTable(TableIdentifier.of("db", "orders"));

FlinkSink.forRowData(dataStream)
  .table(icebergTable)
  .tableLoader(TableLoader.fromCatalog(catalog, TableIdentifier.of("db", "orders")))
  .writeParallelism(4)
  // For upsert, define primary key in table DDL
  .build();

7. Exactly-Once & Checkpoint Tuning: Hard-Won Lessons

Flink writes Iceberg via two-phase commit (2PC): data lands in temp files at checkpoint; atomic commit on full ACK. Production checkpoint failures are common. Key tuning parameters (Flink 1.15+):

Configuration conf = new Configuration();
// 1. Unaligned checkpoints: solve backpressure-induced timeouts, but consume more memory
conf.setBoolean("execution.checkpointing.unaligned", true);
// 2. Incremental checkpoints: with RocksDB, upload only state deltas, drastically reducing HDFS/S3 I/O
conf.setBoolean("execution.checkpointing.incremental", true);
// 3. Timeout and interval
conf.set("execution.checkpointing.timeout", Duration.ofMinutes(10));
conf.set("execution.checkpointing.min-pause", Duration.ofSeconds(30));

env.configure(conf);

Gotcha: Unaligned checkpoints spike Network Memory. If TaskManagers OOM, increase taskmanager.memory.network.fraction or set a fixed minimum (e.g., 1.5GB).

8. Technology Selection: Kafka, Hudi, or Iceberg?

Don't expect one component to rule all. Kafka: millisecond latency, ideal for real-time risk control/recommendation buffering; weak on OLAP and Update/Delete. Iceberg vs Hudi: both are lake formats. Hudi excels at high-frequency upserts, but Iceberg's lighter architecture, hidden partitioning, and superior Spark/Trino query performance currently win. Our pattern: "hybrid doubles" — ultra-real-time data to Kafka, reporting/BI data via Flink CDC into Iceberg, federated queries via Trino.

9. Metadata & Lineage: Prevent Data Swamps

Without governance, data lakes become swamps. Use REST Catalog to decouple metadata from business, enabling Apache Atlas/DataHub integration. For lineage, adopt OpenLineage : via Flink Listeners, extract DAG on job start/stop, push to Marquez. Upstream MySQL schema changes instantly reveal impacted downstream reports in DataHub — no more shouting in chat groups.

10. Production Survival Guide

Now mostly on K8s with Flink Kubernetes Operator.

10.1 Memory Tuning

Network Memory : Critical for unaligned checkpoints and heavy shuffles — allocate generously.

Managed Memory : Must be sufficient for RocksDB state backend; otherwise frequent disk spills kill performance.

10.2 Backpressure Diagnosis

Check Flink Web UI BackPressure panel. HIGH = downstream blocked. Often caused by slow Iceberg writes (many small files → slow commits). Fix: increase sink parallelism or raise Iceberg commit.interval to reduce commit frequency.

10.3 Self-Healing

Hardware fails. Configure restart-strategy in FlinkDeployment CRD with exponential backoff. K8s restarts pods; Flink Operator restores from latest checkpoint. As long as checkpoints persist, business impact is zero.

# K8s CRD snippet
spec:
  job:
    upgradeMode: savepoint
    flinkConfiguration:
      restart-strategy.type: exponential-delay
      restart-strategy.exponential-delay.initial-backoff: 10s
      restart-strategy.exponential-delay.max-backoff: 5m

Key Takeaways

Real-time data lakes are half technology selection, half engineering execution. Flink CDC 3.0's whole-DB sync and schema evolution dramatically reduce pipeline complexity; Iceberg's hidden partitioning and snapshots give lakes true warehouse analytics capability. But don't worship "silver bullets." In production, checkpoint tuning, fine-grained memory control, and backpressure debugging — the gritty work — determine stability. Respect internals, watch metrics, avoid superstition: that's the data engineer's way.