SeaTunnel: Design Goals, Current Status, Architecture, and Future Roadmap

With the rapid development of big data technologies, a growing variety of databases, data warehouses, and data lakes have emerged, creating a critical need for efficient data integration across diverse sources and targets. Apache SeaTunnel, a next‑generation data integration platform, addresses this need by offering a simple‑to‑use, distributed, and highly scalable solution that supports high‑throughput, low‑latency data synchronization.

Design Goals

Support a wide range of data sources and sinks.

Provide fast synchronization with simple configuration.

Offer both batch and streaming (CDC) capabilities.

Ensure high performance, precise exactly‑once semantics, and low latency.

Maintain an active community and extensive user base.

Current Status

More than 50 connectors (sources and sinks) are available, including ClickHouse, Doris, and others, with additional connectors under development.

Unified batch‑and‑stream processing: a single connector can operate in either mode via configuration, supporting both pure streaming (Flink) and micro‑batch (Spark) models.

Multi‑engine support: connectors run on Flink, Spark, or the dedicated SeaTunnel Engine, with a translation layer that decouples connectors from the underlying engine.

High throughput, precise exactly‑once processing, and low latency are achieved through parallelized connectors, two‑phase commit, and snapshot mechanisms.

The community has grown rapidly, with thousands of users and active development.

Overall Design

Engine‑Independent Connector API that abstracts away engine specifics.

Connector Translation layer that maps SeaTunnel connectors to engine‑specific implementations.

Core connector types: Source, Transform, and Sink.

Support for parallel and coordinated source connectors, enabling CDC and dynamic partition discovery.

Sink API with writer, state storage, two‑phase commit, and global commit strategies (driver, worker, or per‑task).

Table & Catalog API for simplified, visual job configuration and metadata management.

Usage Flow

Configuration file is provided; SeaTunnel parses it, creates a job, and submits it.

Source connectors read data (with parallelism and coordination as needed).

Transform connectors standardize data.

Sink connectors write data with exactly‑once guarantees.

Future Roadmap

Double the number of supported connectors to over 80.

Release SeaTunnel Web for visual job management, supporting both programmatic and guided configurations, with internal and third‑party scheduling.

Introduce SeaTunnel Engine to reduce JDBC connections and binlog duplication, enable pipeline isolation, shared threads, and richer monitoring metrics.

The presentation concludes with thanks to the audience.