What is Debezium? Overview, Architecture, and Features

What is Debezium?

Debezium is a distributed platform that turns your existing databases into event streams, allowing applications to see every row‑level change and react instantly. Built on Apache Kafka, it provides Kafka‑Connect compatible connectors to monitor specific DBMSs, recording change history in Kafka logs so applications can consume events reliably even after restarts.

Debezium Architecture

The most common deployment uses Apache Kafka Connect, which provides a framework and runtime for implementing and operating connectors.

Source connectors such as Debezium ingest data into Kafka.

Sink connectors propagate data from Kafka topics to other systems.

The diagram below shows a CDC pipeline based on Debezium:

Besides the Kafka brokers, Kafka Connect runs as a separate service. Debezium connectors for MySQL and Postgres capture changes by connecting to the source databases using the binlog for MySQL or logical replication for Postgres.

By default, changes from a captured table are written to a corresponding Kafka topic, but topic names can be adjusted with Debezium’s topic‑routing SMTs to use custom names or consolidate multiple tables.

Once change events are in Kafka, various connectors from the Kafka Connect ecosystem can stream them to other systems such as Elasticsearch, data warehouses, analytics platforms, or caches like Infinispan. Depending on the sink connector, you may apply Debezium’s ExtractNewRecordState SMT to propagate only the “after” structure.

Embedded Engine

An alternative is to run Debezium as an embedded engine inside a custom Java application, bypassing Kafka Connect. This is useful for internal consumption of change events without deploying a full Kafka and Connect cluster, or for forwarding changes to other messaging systems like Amazon Kinesis. Example code is available in the sample repository.

Debezium Features

Debezium is a set of source connectors for Apache Kafka Connect that use change data capture (CDC) to obtain changes from various databases. Compared with polling or dual‑write approaches, log‑based CDC provides:

Guaranteeing capture of all data changes.

Very low latency (milliseconds for MySQL or Postgres) while avoiding CPU overhead of frequent polling.

No need to modify the data model (e.g., adding “last_updated” columns).

Ability to capture deletions.

Ability to capture old record state and metadata such as transaction IDs and the query that caused the change (depending on database capabilities and configuration).

Key CDC capabilities are exposed through a range of options:

Snapshots: optional initial snapshot of the current database state when the connector starts and logs are no longer available.

Filters: whitelist/blacklist filters to select schemas, tables, and columns.

Masking: hide values in specific columns, e.g., sensitive data.

Monitoring: most connectors expose JMX metrics.

Various message transformations: e.g., routing, extracting new record state, routing events from transactional outbox tables.

For a full list of supported databases and detailed connector configuration, refer to the connector documentation.

Original source: https://debezium.io/documentation/reference/0.10/features.html

Article: https://pub.intelligentx.net/technical-architecture-cdc-capture-data-changes-debezium-introducuction