Real-Time Data Warehouse Evolution: Flink Writes, Paimon Stores, Doris Queries, One Platform Manages

The article presents a lightweight real-time data warehouse (RT-DWH) architecture built around four core components: Flink 2.0+ for continuous streaming computation and reliable writes, Apache Paimon as the unified lakehouse storage layer, Apache Doris for low-latency interactive queries, and a custom RT-DWH management platform that orchestrates tasks, metadata, data quality, access control, and monitoring.

01 Why Separate Query from Flink

Flink excels at long-running streaming jobs — reading binlogs, handling out-of-order events, maintaining state, and guaranteeing consistency via checkpoints. However, running ad-hoc analytical queries through Flink SQL Gateway introduces several problems:

Each query creates a session, submits an operation, and waits for resource scheduling.

Query tasks compete with long-running CDC jobs for slots.

High-concurrency short queries amplify Flink's scheduling and startup overhead.

User cancellation, timeouts, pagination, and result export require extra query-lifecycle management.

Query peaks can back-pressure the real-time write pipeline.

The root cause is undifferentiated compute responsibilities . In the new architecture, Flink continues to own real-time writes, stream processing, and table maintenance, while Doris reads the same Paimon data via Paimon Catalog to serve developers, analysts, BI tools, and APIs. The key benefit: real-time pipeline stability is decoupled from ad-hoc query concurrency .

02 Core Architecture: One Paimon Dataset, Two Compute Engines

The architecture comprises five layers:

Layer 1: Data Sources

Primary sources are MySQL and PostgreSQL. For small-to-medium scale, Flink CDC reads binlog/WAL directly without mandatory Kafka. Kafka is only added when traffic shaping, multi-consumer decoupling, or long-term event replay are needed.

Layer 2: Flink 2.0+ Real-Time Compute

Flink handles schema discovery, CDC ingestion, field mapping, deduplication, joins, aggregations, and Paimon writes. Flink 2.0 removed legacy Source/Sink APIs; connectors must be Flink 2.x compatible. Paimon 2.0 provides separate connector packages for Flink 2.0, 2.1, 2.2. The deployment scripts use paimon-flink-2.2 with Paimon 2.0.0; the article uses "Flink 2.0+" to denote the 2.x evolution line.

Layer 3: Paimon Lakehouse Storage

Paimon serves as both table format and real-time update store. Data files reside on MinIO, S3, HDFS, or shared filesystems; Catalog stores databases, tables, schemas, snapshots, partitions, and primary keys. Two lightweight catalog options:

Filesystem Catalog : minimal components, suitable for single-environment small teams.

JDBC Catalog : reuses MySQL/PostgreSQL for metadata, better for unified management-platform retrieval.

Note : Doris 4.1+ supports Paimon JDBC Catalog but marks it experimental. For production stability, prefer Filesystem or HMS Catalog; if JDBC Catalog is required, lock Doris version and run dedicated compatibility tests.

Layer 4: Doris Interactive Query

Doris reads table metadata and data files via Paimon Catalog, leveraging vectorized execution, parallel scan, and query optimizer. No data replication into Doris internal tables is needed. Example query:

SELECT
  shop_id,
  DATE_FORMAT(pay_time, '%Y-%m-%d') AS pay_date,
  SUM(pay_amount) AS gmv
FROM paimon_lake.dwd.dwd_order_detail
WHERE pay_time >= NOW() - INTERVAL 7 DAY
GROUP BY shop_id, DATE_FORMAT(pay_time, '%Y-%m-%d')
ORDER BY gmv DESC
LIMIT 100;

Doris is positioned as read-only query and query acceleration . Writes, compaction, and snapshot cleanup remain with Flink and Paimon to avoid multiple engines modifying table state concurrently.

Layer 5: RT-DWH Management Control Plane

The platform does not act as a compute engine but orchestrates the stack:

Manage data sources, CDC tasks, and Flink job lifecycles.

Maintain Paimon Catalog, schemas, snapshots, and warehouse layering.

Route ad-hoc queries safely to Doris.

Enforce read-only validation, RBAC, rate limiting, timeouts, and max-row limits.

Manage compaction, snapshot expiration, and orphan-file cleanup.

Collect job, checkpoint, lag, throughput, and query latency metrics.

Centralize quality rules, alerts, auditing, and system configuration.

Recent additions include Doris JDBC connection pool, automatic Paimon Catalog initialization, catalog directory reading, connection health checks, and runtime configuration. Query responses return execution engine, catalog, database, and trace ID for end-to-end tracing.

"Lightweight" does not mean stuffing all capabilities into one process; it means letting each component do what it does best and keeping complexity inside the management platform.

03 Write Path: Flink CDC to Paimon Tables

A sync task follows this core flow (pseudocode):

function deployCdcTask(taskConfig):
  source = loadDatasource(taskConfig.sourceId)
  target = loadPaimonCatalog(taskConfig.catalogId)
  assert source.type in [MYSQL, POSTGRESQL]
  assert target.warehouse is reachable
  schemas = introspectSourceTables(source, taskConfig.tableMappings)
  for each schema in schemas:
    validatePrimaryKey(schema)
    targetSchema = mapTypesToPaimon(schema)
    ensurePaimonTable(target, targetSchema)
  flinkSql = generateCdcSql(source, target, schemas,
    taskConfig.startupMode, taskConfig.checkpointInterval)
  previewToUser(flinkSql)
  jobId = submitToFlink(flinkSql)
  saveTaskState(taskConfig.id, SUBMITTING, jobId)
  while task is active:
    status = queryFlinkJob(jobId)
    if status == RUNNING:
      updateMetrics(checkpoint, lag, throughput)
    else if status == SUSPENDED:
      calibrateTaskState(PAUSED)
    else if status == NOT_FOUND:
      calibrateTaskState(FINISHED)
    else if cluster temporarily unreachable:
      keepCurrentState()

The critical challenges are three consistencies:

Structural consistency : source fields, Paimon schema, and Doris read types must be compatible.

State consistency : platform-reported state must closely match Flink job's actual state.

Recovery consistency : pause/resume must be built around checkpoints, savepoints, and job IDs with full audit trail.

04 Query Path: Doris as Unified Paimon Query Entry

The query workbench must not pass user SQL directly to Doris; it first enforces security, resource, and audit boundaries:

function executeAdhocQuery(request, currentUser):
  sql = normalize(request.sql)
  assert currentUser has QUERY_PERMISSION
  assert sql is singleStatement
  assert firstKeyword(sql) in [SELECT, SHOW, DESCRIBE, EXPLAIN, WITH]
  assert sql does not contain writeOrDdlKeyword
  queryPlan = resolvePaimonCatalog(sql)
  assert currentUser canAccess(queryPlan.databases, queryPlan.tables)
  maxRows = min(request.maxRows, SYSTEM_ROW_LIMIT)
  timeout = min(request.timeout, SYSTEM_TIMEOUT)
  queryId = createQueryHistory(user=currentUser, sql=sql, status=RUNNING)
  try:
    connection = dorisPool.acquire()
    connection.setQueryTimeout(timeout)
    result = connection.execute(sql)
    page = fetchAtMost(result, maxRows + 1)
    return {
      rows: page.first(maxRows),
      truncated: page.size > maxRows,
      queryId: queryId
    }
  catch error:
    recordFailure(queryId, concise(error))
    throw userFriendlyError(error)
  finally:
    release(connection)
    recordDuration(queryId)

Doris exposes MySQL-compatible protocol; RT-DWH backend accesses Doris FE via connection pool. Frontend retains catalog tree, SQL autocomplete, query history, cancel, export — but execution engine switches from Flink SQL Gateway to Doris.

On startup, the platform runs CREATE CATALOG IF NOT EXISTS to register the existing Paimon JDBC Catalog in Doris. Before each query it executes SWITCH catalog and USE database, sets timeout and trace ID. Catalog browsing uses SHOW DATABASES, SHOW TABLES, DESCRIBE dynamically, no longer relying on a metadata copy in the management DB.

Four protective layers prevent query layer from dragging down the lakehouse:

Row-count protection : different limits for UI preview vs. file export.

Time protection : hard timeout for interactive queries; large offline queries go to separate queue.

Concurrency protection : per-user/role/tenant query concurrency limits.

Resource protection : Doris Workload Groups isolate ad-hoc, BI, and API workloads.

05 Why Query Paimon Directly Instead of Syncing to Doris Internal Tables

Periodically importing Paimon data into Doris internal tables yields good query performance but reintroduces data duplication, latency, and consistency-of-definition issues. Direct Paimon query advantages:

Single dataset : Flink writes once, Doris reads the same data.

Single catalog : schema, snapshots, partition boundaries stay unified.

Low migration cost : no extra sync pipeline to build first.

Progressive optimization : hot spots can later be materialized into Doris internal tables or materialized views on demand.

Openness preserved : Paimon data remains accessible to Flink, Spark, Trino, etc.

"Zero-copy" ≠ "zero-cost". Primary-key tables may need merge reads; fragmented data files amplify scan overhead; catalog cache affects new snapshot/schema visibility. Production must monitor:

Choose appropriate bucket, merge engine, and read-optimized strategy for primary-key tables.

Run regular compaction to control small-file count.

Set Doris Paimon metadata cache TTL per freshness requirements.

Observe table evolution and file distribution via system tables (snapshots, files).

Validate Doris read support for each new Paimon 2.0 data type and table feature.

Recommended two-tier query strategy:

Direct Paimon Query via Doris Paimon Catalog — for detail exploration, low-latency reports, unified-definition queries.

Hot Data into Doris Internal Table via incremental import or scheduled build — for high-concurrency APIs, fixed aggregations, sub-second dashboards.

Start with Paimon Catalog to solve "unified query", then accelerate only what real query profiles justify — more aligned with lightweight principles than copying all data upfront.

06 Practical Lightweight Deployment

Minimum viable stack for validation or small-scale production:

1 RT-DWH backend + 1 frontend

1 MySQL (management data + optional Paimon JDBC Catalog)

1 Flink JobManager + 1–2 TaskManagers

1 MinIO or shared storage cluster (Paimon warehouse)

1 Doris FE + 1–3 Doris BEs

Optional Prometheus, Grafana, alerting channels

Phase 1: verify three main paths without full HA: 1. MySQL/PostgreSQL → Flink CDC → Paimon 2. Doris → Paimon Catalog → query results 3. RT-DWH → Flink/Paimon/Doris → unified management

Post-production: incrementally add Flink HA, multi-replica Doris FE, object-storage HA, catalog backup, multi-instance management platform based on failure-domain analysis.

07 Version Compatibility Matrix

Stability depends on actual version compatibility, not co-appearance on a diagram. Lock and validate before go-live:

Flink 2.x : Must confirm CDC Connector, Paimon Connector, SQL Gateway, Java version.

Paimon 2.x : Must confirm Catalog type, table format, primary-key table modes, new data types, filesystem plugins.

Doris : Must confirm Paimon Reader version, Catalog type, Deletion Vector, system tables, Time Travel support.

Object Storage : Must confirm Endpoint, region, path-style, access keys, directory permissions.

Management Platform : Must confirm JDBC driver, connection pool, SQL allowlist, timeouts, auditing, state calibration.

Critical pitfall : Paimon 2.0 providing Flink 2.x connectors does not mean all Doris versions can unconditionally read every Paimon 2.0 feature. Build a minimal regression suite covering append-only tables, primary-key tables, upserts/deletes, schema evolution, snapshots, compaction, partition pruning, and common complex types. Only when these pass is the version combo truly production-ready.

Conclusion: Lightweight Essence Is Clear Boundaries

A platform becomes "heavy" not from component count but from duplicated responsibilities. Paimon + Flink 2.0+ + Doris offers a cleaner real-time data warehouse path: Flink sustains continuous computation and reliable writes; Paimon holds unified, open data assets; Doris delivers fast human-facing queries; RT-DWH connects tasks, metadata, quality, permissions, and operations. This path doesn't demand big-bang adoption — teams can start with single-node validation and scale compute/storage nodes only when data volume, query pressure, and reliability requirements genuinely grow.

A good lightweight architecture isn't about fewer features; it's about making sophisticated capabilities usable at lower cognitive cost.

References

Apache Paimon 2.0: Flink version connector download notes

Apache Flink 2.0 Release Notes

Apache Doris: Paimon Catalog

Apache Doris: Doris & Paimon Best Practices

Apache Paimon: Catalog Concepts & Types

Source code repository: https://github.com/liuzm/rt-dwh-mgmt.git