Designing a Management Backend for Real‑Time Data Warehousing with Flink + Paimon (Part 1)

Why the system is needed

Flink + Paimon is popular for real‑time data warehouses in small‑to‑medium teams, but operational practices remain script‑centric, causing concrete problems:

Flink jobs are started or stopped via curl to the REST API or by SSH‑ing to a server and running the Flink CLI; there is no web UI, so debugging requires log inspection and is inefficient.

Paimon metadata is stored in a few MySQL tables of the JDBC Metastore; to know the number of tables, layer distribution, or data size one must query MySQL directly, which lacks visualization and indexing.

Ad‑hoc SQL queries must connect to the Flink SQL Gateway or HiveServer2, and accidental DROP statements can delete tables.

Job failures and latency breaches are discovered only through manual checks or business‑side complaints.

Data‑quality checks are occasional; issues are often detected too late.

Impact analysis for schema changes relies on memory of lineage, which is unreliable.

Although Flink and Paimon provide the necessary capabilities, management remains at the script level. The proposed real‑time data‑warehouse (RT‑DWH) platform fills this gap by offering a unified web‑based management console that consolidates daily operations.

What the system can do

In a single sentence: it centralizes data‑source ingestion, table maintenance, and routine operations of a real‑time data warehouse.

Core modules

Sync Task Management – Orchestrates Flink jobs with three task types (CDC sync from MySQL/PostgreSQL to Paimon, ETL transformation via Flink SQL, materialized view for pre‑aggregation). It follows a state machine

Draft → Submitting → Running → Paused / Failed / Completed

. When a task is paused, a Savepoint is automatically created; recovery restarts from the Savepoint to guarantee Exactly‑Once semantics. Flink 2.x’s pure SQL Gateway submission abstracts the underlying submission method.

Metadata Management – Automatically pulls database, table, and column information from Paimon’s JDBC Metastore, infers layer prefixes ( ods_, dwd_, dws_, ads_) to categorize tables, and allows business‑level annotations on tables and fields for better readability.

Ad‑hoc Query – Provides a secure SQL editor with syntax highlighting. Only SELECT, SHOW, DESCRIBE, EXPLAIN, and WITH statements are accepted; DROP and DELETE are rejected. Query history is recorded and results can be exported to Excel.

Data Quality – Implements five rule types (null‑rate, uniqueness, row‑count fluctuation, value‑range, freshness). Rules can be triggered manually or on a schedule, and violations generate automatic alerts.

Alert Notification – Supports DingTalk webhook, WeChat Work webhook, and email channels. Alerts fire on task failure, latency breach, or quality anomaly, and each channel can have distinct notification strategies.

Table Maintenance – Addresses Paimon’s small‑file accumulation and snapshot bloat: minor and full compaction, snapshot expiration (retain the latest N snapshots), isolated file reclamation, and batch actions per layer.

Technical selection

Frontend : React 18 + Umi Max + Ant Design Pro – standard enterprise‑backend stack with a rich component ecosystem.

Backend : Java 17 + Spring Boot 3.3 + JPA – LTS version; JPA fits CRUD‑heavy, SQL‑intensive scenarios.

Authentication : JWT + Spring Security – stateless authentication simplifies containerized deployment.

Database : MySQL 8.0 – stores metadata and aligns with the Paimon Metastore technology stack.

Scheduler : Quartz – synchronizes Flink cluster state and integrates well with Spring Boot.

Deployment : Docker Compose + Nginx – one‑click orchestration; Nginx reverse‑proxies and serves static assets.

The difficulty lies not in the technology stack itself but in integrating Flink’s REST API and synchronizing Paimon metadata, which contain many pitfalls.

Target users

Data‑warehouse operators – manage Flink jobs, monitor health, handle alerts.

Data developers – create tasks, browse metadata, write SQL.

Data analysts – self‑serve queries and reports without relying on operators.

Team leads – view dashboards for overall health.

Because role requirements differ greatly, the platform implements role‑based permission tiers (admin, develop, visitor) that filter menus and pages accordingly.

Next article will discuss architecture details, module responsibilities, data flow, and Flink dual‑mode submission.