Ziroom's One-Click Lake Ingestion: StreamPark + Paimon for 4400+ MySQL Tables

Background and Traditional Architecture Challenges

Ziroom's long-term rental business generates data from rental, order, housing, contract, finance, and IoT systems across 4,400+ MySQL tables, 8,000+ Hive ETL tasks, and ~50 TB daily new data. Tables are split into 4,000+ T+1 low-latency tables and 400+ ~10-minute high-latency tables.

The legacy architecture used scheduled full pulls via Hive JDBC Handler for low-latency tables, and for high-latency tables: MySQL → Canal → Kafka → Flink → HDFS/Hive external incremental tables → Airflow scheduled merge → Hive target tables. This worked but created five structural problems:

Multiple full-load tasks hitting source databases simultaneously, causing I/O, bandwidth, and connection pressure.

Incremental and full data stored separately, requiring extra merge, deduplication, and scheduling jobs.

Source schema changes forced synchronized updates across collection scripts, transformation logic, target tables, and downstream SQL.

Long chain (Canal, Kafka, Flink, HDFS, Airflow) increased failure diagnosis and operational complexity.

Per-table task configuration and deployment amplified job management, resource allocation, and change management costs.

New Architecture: StreamPark + Flink + Paimon

The revised pipeline: MySQL → Flink CDC → Paimon → Hive/downstream consumption. A data access platform handles requests, OA approval, then calls StreamPark APIs to create/start Flink sync jobs. Paimon CDC Action reads MySQL snapshots and binlogs directly, writing to Paimon tables.

Key clarifications:

Paimon CDC Action does not eliminate CDC; it lets Flink read MySQL snapshots and binlogs directly, removing Canal/Kafka transit for this specific path. Initial snapshots and continuous binlog consumption with checkpoint/state recovery remain.

StreamPark is not a CDC engine or storage layer; it provides centralized Flink job management (create, build, deploy, run) and API integration with external systems. Actual data reading, CDC progress, and Paimon writes are done by Flink jobs and connectors.

Core Component Roles

Apache Paimon : Lake storage, table metadata, snapshot management, stream/batch access. Its CDC Action continuously writes supported source changes into Paimon tables.

Apache Flink : Source snapshot reading, binlog capture, data processing, target writing. Semantics depend on Source, Sink, Checkpoint, and Connector configuration.

Apache StreamPark : Centralized Flink job management and automated deployment, enabling integration with enterprise data access platforms.

Hive : Downstream query entry for lake tables in this case. Direct Paimon reads depend on Paimon/Hive versions, Catalog config, and access components; not universally compatible.

Deep Dive: Paimon CDC Actions and Schema Evolution

mysql_sync_database vs mysql_sync_table

mysql_sync_database

: Syncs selected MySQL tables into one Paimon database, each source table typically mapping to one target table. Reduces per-table Flink application overhead. mysql_sync_table: Syncs one or multiple source tables into a single Paimon table, suited for merge scenarios with compatible structure and key semantics.

Paimon docs note database sync only syncs tables with primary keys. Pre-flight checks for primary keys, type compatibility, name conflicts, and target schema are still required. Resource usage does not scale linearly with job count reduction; a single job still contains multiple sources, sinks, and state, with actual usage depending on parallelism, table count, data rate, and state size.

Example command (simplified, version-specific):

<FLINK_HOME>/bin/flink run \
  /path/to/paimon-flink-action-<version>.jar \
  mysql_sync_database \
  --warehouse <warehouse-path> \
  --database <target-database> \
  --including_tables 'order|contract|house.*' \
  --mysql_conf hostname=<mysql-host> \
  --mysql_conf username=<username> \
  --mysql_conf password=<password> \
  --mysql_conf database-name=<source-database>

Schema Evolution Boundaries

Paimon CDC Action auto-creates target tables and updates target schema on supported schema change events, reducing manual propagation for added fields and compatible type changes. However, it does not handle all DDL changes:

Add column: target table adds corresponding column.

Compatible type widening: target column type can evolve within supported range.

Drop column: does NOT auto-drop target column.

Rename column: typically treated as new column; old column may remain.

Primary key, partition key, other constraints: cannot assume automatic migration.

Unsupported changes (drop, rename, PK changes, downstream metric impacts) still require data contract and change review processes. Auto evolution solves supported field change propagation, not full schema governance.

StreamPark API Automation: From Callable to Reliable

At thousands of jobs, manual CLI/Web deployment cannot meet batch, standardized, auditable needs. StreamPark provides unified job management entry. The pattern abstracts to:

Data Access Request → Approval → Parameter Validation → StreamPark API → Job Build → Deploy & Start → Status Callback

Templates reuse validated job definitions; parameters specify source DB, target DB, table filters, runtime config. Platform auto-creates jobs post-approval, reducing manual deployment and communication wait.

API-ification ≠ production reliability. Required capabilities:

Idempotency against duplicate approval events.

Compensation for build-success but deploy-failure.

Prevent duplicate consumption of same table by multiple jobs.

Determine when job finishes initial snapshot and enters steady incremental phase.

Sync new source tables into scope while preserving historical completeness.

Authorization, audit, rate limiting, version management.

Production Practices

Group by Business Domain, Not Blind Merge

For thousands of tables, database sync Action reduces job count but putting all tables in one job isn't optimal. Group by business domain, source instance, data volume, update frequency, SLA. Considerations:

Per-batch table count vs source scan pressure.

Per-job parallelism, state size, checkpoint load.

Blast radius on job failure.

Presence of keyless tables, incompatible fields, duplicate names.

Need for independent sync SLAs per data tier.

Goal: balance job count, resource utilization, fault isolation, maintenance complexity.

Schema Evolution in Governance, Not Connector-Only

Auto evolution reduces manual ops but shouldn't let arbitrary source changes hit all downstream. Recommended tiered change mechanism:

Compatible adds and supported type widenings: auto-evolve after automated validation.

Drop column, rename, PK change, partition strategy change: compatibility assessment required.

Changes affecting downstream reports, data services, metrics: notify consumers, verify.

New columns potentially introducing sensitive data: sync permission and compliance checks.

API Automation State Machine

HTTP success ≠ job online. Design unified state machine:

Approved → Param Validation → Create Job → Build Success → Deploy Success → Running → Initial Snapshot Complete → Steady Sync

Each step failure logs error, timestamp, retry count; resume from explicit state, not blind recreate. Unique identifiers per source table/sync group prevent duplicate competing jobs. Stop/restart/upgrade must verify Checkpoint/Savepoint compatibility.

Leverage Lake Snapshots to Reduce Repeat Full Reads

Once data reliably enters Paimon via CDC, downstream tasks can read committed lake snapshots instead of rescanning source DB for repeated analytical results. Two phases:

Initial lake ingestion: may still need MySQL snapshot for initialization.

Subsequent consumption: read Paimon committed data per business need, reducing source revisits.

Paimon Tag retains specified snapshots for reproducible batch reads or fixed historical versions. Tag does NOT inherently lower storage cost; it preserves referenced snapshot files, potentially extending file lifetime. Cost benefit comes from reduced duplicate full pulls, recomputation, and extra copies — not from creating Tags alone.

Remove Middleware by Responsibility, Not Shortest Path

Canal, Kafka, Airflow have clear use cases. If sole goal is MySQL CDC → Paimon, Flink CDC Action can drop Canal/Kafka and incremental merge jobs. But if Kafka provides multi-consumer sharing, buffering, async decoupling, or independent replay, evaluate impact of removal. Similarly, removing Hive incremental merge doesn't eliminate all scheduling needs: task dependencies, initial snapshot monitoring, backfill, data quality checks, failure recovery may still require orchestration. Principle: remove duplicate responsibilities, not components at expense of decoupling, recoverability, observability.

Performance, Cost, Selection Criteria

Structural Benefits

Reduce duplicate data processing: continuous CDC into lake cuts some repeat full reads and downstream merges, but initial snapshot, continuous log read, lake maintenance costs remain.

Reduce independent jobs and middle hops: database-level sync Action manages multiple tables in one Flink job; StreamPark API cuts manual create/deploy effort. Actual resource gains depend on parallelism, table count, job load.

Simplify ops and change flows: auto table creation and supported Schema Evolution lower manual ops for some field changes, but incompatible changes and downstream schema management persist.

Selection Fit

Prioritize Paimon + Flink CDC + StreamPark when:

Many MySQL tables needing continuous lake ingestion.

Clear latency tiers requiring both initial full and incremental sync.

Team has Flink and Paimon expertise to maintain CDC jobs and lake tables.

Enterprise wants unified data access platform integrating approval, job creation, deployment, run monitoring.

Verify target versions, PK requirements, source DDL change scope, job recovery, downstream read compatibility. For small scale, few sources, simple sync logic, existing integration platforms may be more economical. For existing Kafka CDC ecosystems needing multi-downstream reuse, keeping Kafka may be more rational. Selection isn't "more tables → delete all components" but "more tables → need standardized sync patterns, unified job management, automated lifecycle governance."

Production Issues and Troubleshooting

Issue 1: Single Table Failure in Full-DB Sync Job Affects Others

Symptom : One table's schema incompatibility, source anomaly, or write failure causes whole job to error or lag. Cause : Multiple tables share job runtime resources and partial failure domain; single-table issue can trigger job failure or resource contention depending on sync mode, connector implementation, exception handling. Resolution : Check Flink logs to locate failing table, data type, failure stage. Verify source PK, column types, target schema, sync filters. For unfixable tables, split out into independent sync. Re-partition sync groups by table count, data rate, recovery needs. Prevention : Group by business domain and risk level; monitor per-table sync progress and error state. Don't maximize tables per job at expense of blast radius.

Issue 2: Source Column Added, Lake Table Updated, Downstream Query Fails

Symptom : MySQL adds column, Paimon target shows it, but downstream SQL/Hive view/BI model breaks. Cause : Source change within Paimon CDC Action's supported Schema Evolution, but downstream query models not updated; or target column type/constraint incompatible. Resolution : Confirm source DDL type is supported. Inspect actual Paimon target schema, not just source. Check Hive Catalog, cached metadata, downstream SQL column references. For drops, renames, type changes, define compatibility and migration steps. Prevention : Establish schema change notification and downstream validation flow. Separate auto-compatible changes from those requiring manual review.

Issue 3: Midnight Full Pulls Still Stress Source DB

Symptom : MySQL I/O, bandwidth, connections spike during full sync. Cause : Some data still relies on scheduled full pulls; new table initialization triggers large snapshot reads; failure recovery causes re-initialization. Resolution : Distinguish initial snapshot, continuous binlog consumption, downstream lake reads. Measure per-table snapshot read volume, concurrency, duration. For stable lake data, evaluate downstream reading from Paimon committed snapshots. Set concurrency limits, scheduling windows, DB protection thresholds for initial snapshots. Check for abnormal restarts or repeated initialization when incremental lags. Prevention : Define source DB resource needs for snapshot init vs continuous CDC phases; don't assume "source only outputs binlog" applies to all phases.

Issue 4: Growing Job Count Causes Duplicate Tasks or State Inconsistency in Auto-Deploy

Symptom : Same request creates duplicate jobs, or build succeeds but start fails while platform shows completed. Cause : API timeout retries, duplicate approval consumption, non-idempotent deploy, incomplete state sync between platform and StreamPark. Resolution : Stable business ID and target table unique constraint per access request. Audit API call logs, application IDs, actual job run states. Independent state records and timeouts for build, deploy, start. Idempotent retry and compensation for failed stages. Prevention : Build automation platform as explicit state machine workflow, not chained API calls. Provide platform-side permissions, audit, alerting, manual intervention.

Key Takeaways

Full-database sync reduces repetitive task management cost, not all sync costs. Initial snapshot, binlog consumption, checkpoint, lake maintenance persist.

Schema Evolution has clear boundaries: auto-handles some adds and compatible type changes, but cannot replace data contracts and downstream governance.

StreamPark API is job management capability, not CDC engine. One-click lake ingestion requires access platform to handle approval, param validation, idempotency, compensation, state management.

Paimon Snapshot and Tag serve data lifecycle management. Tag retains specified snapshots for versioned reads, not an inherent storage cost reducer.

Shortening pipeline must be based on responsibility assessment. Retain Canal, Kafka, Airflow if they still own irreplaceable functions.

Bottom line : True one-click lake ingestion isn't merely deleting middleware; it's turning data access into a governable, monitorable, recoverable platform capability through standardized CDC, controlled schema evolution, batch table management, and API-driven job lifecycle.