Boosting Flink CDC to Hologres: High‑Performance Data Sync Optimization Techniques

Abstract: This talk covers Flink CDC + Hologres high‑performance data synchronization optimization, presented by Alibaba Cloud senior technical expert Hu Yibo, divided into three parts: write optimization, consumption optimization, and future outlook.

01 Hologres Overview

Hologres is a real‑time data warehouse offering integrated OLAP and serving capabilities with millisecond‑level write latency, high QPS, PG‑compatible SQL, and support for vector search. It allows simultaneous OLAP analysis and serving on the same table with isolated compute resources.

02 Hologres Connector

The Hologres connector supports all Flink features, including dimension tables with million‑level point queries and result tables with real‑time upserts and DDL synchronization. It reads full data and incremental binlog, and integrates with Flink’s Catalog interface.

03 Hologres Write Optimization

Write optimization includes buffering queues, hash‑based sharding, and a connection pool to increase throughput. Aggressive mode triggers immediate commits when a connection is idle, reducing latency to sub‑second levels. Fixed‑frontend threading and sdkMode: jdbc_fixed increase concurrency and lower connection costs.

Batch INSERTs are enhanced with COPY‑style streaming ( STREAM_MODE=true) to achieve up to eight‑fold throughput improvement and lower TaskManager memory usage.

Offline write mode with shard‑level locks and repartitioning reduces CPU usage by ~70% when millisecond latency is not required.

04 Hologres Consumption Optimization

Consumption optimization replaces row‑format SELECT with PostgreSQL COPY to improve connection utilization and CPU efficiency. For large data exports, copy operations can be offloaded to serverless resources. Partitioned tables are handled by launching readers per shard, with Fixed mode handling connection limits.

Future Outlook

Future work aims to unify all write paths via COPY, expand schema‑evolution support, and provide full‑incremental CDC without overlap by introducing snapshot reads. Hologres 3.0 will integrate real‑time lakehouse capabilities, dynamic tables, external databases, and AI features.