This article details JD Search's journey from a Storm‑based pipeline to a Flink‑driven architecture backed by Apache Doris, covering business requirements, technical challenges, design trade‑offs, performance optimizations for massive traffic spikes, and future plans for their real‑time OLAP data warehouse.
This article demonstrates how to initialize, broadcast, and dynamically update rule sets in an Apache Flink fraud detection pipeline, using BroadcastProcessFunction and MapState to achieve runtime data partitioning without recompiling, and explains the underlying data exchange patterns such as forward, hash, rebalance, and broadcast.
The article details Apache Iceberg 0.11.0's core enhancements—including partition changes, SortOrder, extensive Flink and Spark integrations, CDC/Upsert support, hash‑based write distribution to reduce small files, and upcoming 0.12.0 roadmap—while providing practical SQL and API examples for data‑lake practitioners.
This article presents Meituan's use of Flink to enable incremental data warehouse production, covering the warehouse architecture, streaming data integration evolution, real-time OLAP applications, platform design, and future directions for unified stream‑batch processing.
This guide walks through setting up Apache Zeppelin as a low‑cost, SQL‑centric development platform for Flink, covering environment preparation, installation, interpreter configuration, execution modes, verification, common pitfalls, dimension‑table joins, custom UDFs, Redis integration, and dual‑stream join techniques.
The article reviews Flink 1.12's major enhancements, including comprehensive SQL capabilities, deep integration with Hive via catalog and streaming support, and a practical code example that demonstrates how to write streaming data into Hive tables while handling partition commits and small‑file merging.
This article provides a comprehensive introduction to Apache Kafka, covering its distributed publish‑subscribe architecture, its key components such as brokers, topics, partitions, producers, consumers, and ZooKeeper, as well as its advantages, drawbacks, storage mechanisms, partition assignment strategies, and reliability guarantees for high‑throughput big‑data streaming.
This article presents Tencent's experience of constructing a real‑time data warehouse by integrating Apache Flink with Apache Iceberg, covering background pain points, Iceberg's table format and capabilities, Flink‑Iceberg streaming and batch processing, practical implementations, and future roadmap for data‑lake acceleration.
iQIYI becomes the first video platform to deliver content meeting the China Ultra‑High‑Definition Video Industry Alliance (CUUVA) HDR standard, enabling devices like Xiaomi 10 Pro, Huawei P30 Pro, iPhone 11 Pro and XS Max to display richer colors, higher contrast, and deeper visual depth, while fostering an open, industry‑wide ecosystem and planning future CUVA HDR live‑streaming support.
Hulu’s Generation Stream study, conducted with Culture Co‑Op and industry experts, surveyed 2,500 U.S. viewers aged 13‑54, revealing that 90% watch TV via streaming, identifying three audience segments—‘Stream Most’, ‘Stream Only’, and ‘Stream Also’—and highlighting how streaming reshapes viewing habits and content expectations.
This article introduces Kafka’s fundamental role as a messaging system, explains topics, partitions, producers, consumers, replicas, consumer groups, and the controller, and explores its cluster architecture, performance optimizations like sequential writes and zero-copy, providing a comprehensive overview for building scalable data pipelines.
This article describes how LinkedIn’s Who Viewed Your Profile feature was originally built on a Lambda architecture, the operational challenges it caused, and the step‑by‑step migration to a streamlined, Samza‑driven, Lambda‑less design that improves performance, reduces maintenance overhead, and retains essential batch capabilities.
This article explains the concepts of streaming output and reactive programming, describes the underlying HTTP chunked transfer, SSE, WebSocket and RSocket protocols, provides code examples, and outlines practical scenarios where end‑to‑end streaming improves performance and user experience.
This guide explains how to integrate Flink 1.12 with Hive using HiveCatalog, covering required dependencies, Blink planner configuration, SQL dialect switching, Hive UDF support, temporal table joins, and provides complete code snippets for a streaming‑batch unified data warehouse solution.
This article presents a detailed collection of 2021 Flink interview questions covering checkpoint mechanisms, watermarks, state backends, join types, fault tolerance, resource configuration, and recent Flink 1.10 features, providing concise explanations and code examples for each topic.
This guide demonstrates how to build a Docker‑based Spark standalone environment, configure Apache Zeppelin to connect to it, and perform data analysis on local CSV files, HDFS, and streaming sources such as Twitter and Kafka, with complete code examples.
This article describes how a fast‑growing company built a layered real‑time data warehouse on Flink, detailing the evolution from a simple 1.0 pipeline to a 2.0 architecture with ODS, DWD and ADS layers, dimension joins, exactly‑once sinks, HDFS partitioning, monitoring, and future improvements.
This article provides a step‑by‑step guide on configuring the MySQL CDC connector for Flink, covering Maven and SQL client dependencies, MySQL user setup, connector options, table creation via SQL and Stream API, key features, common issues, and practical troubleshooting tips.
This article details how to migrate Spark Streaming jobs to Flink Streaming by creating custom SourceFunction and SinkFunction implementations, including a RocketMQ source connector and an HBase sink, with code examples, configuration tips, and discussion of checkpointing and watermark handling.
The MSU World Video Codec Competition highlighted iQIYI’s QAV1 encoder achieving faster speeds and superior compression compared to H.265, while the expanding AV1 ecosystem—bolstered by widespread hardware decoding, platform adoption, and recent QAV1 enhancements such as 8K, HDR, and improved rate‑distortion optimization—promises higher quality video at lower bandwidth costs.
This tutorial introduces Apache Beam’s unified programming model for batch and streaming, explains its core concepts and terminology, compares it with other runners, and walks through a complete Java word‑count example—including dependencies, pipeline construction, transforms, and execution with DirectRunner.
This article explains how Flink enables end‑to‑end machine‑learning workflows through AI Flow, covering the background of Lambda architecture, AI task stages, the advantages of Flink, AI Flow components, AI Graph concepts, integration with Python and TensorFlow, and a real‑world advertising recommendation use case.
This article explains how Apache Flink integrates with data lake architectures, especially using Apache Iceberg as a table format, to enable real‑time streaming ingestion, CDC processing, near‑real‑time lambda architectures, and future enhancements like automatic file merging and row‑level deletes.
This article explains how Flink enables end‑to‑end AI workflows through the AI Flow platform, covering the Lambda architecture background, AI task pipeline stages, the reasons for choosing Flink, AI Flow’s graph model, core services, integration with ML pipelines, and real‑world advertising recommendation use cases.
This article examines Apache Pulsar, an open‑source messaging platform created by Yahoo, compares it with Kafka by outlining Kafka’s common pain points, highlights Pulsar’s multi‑tenant architecture, layered storage, built‑in functions, and security features, and discusses the trade‑offs of each solution.
This article introduces Alink, Alibaba's open‑source machine‑learning platform built on Flink, explains its core algorithms, performance comparison with Spark ML, version‑wise feature evolution, and provides practical quick‑start instructions for both Java (Maven) and Python (PyAlink) users, including data source handling, type conversion components, unified file‑system operations, and an overview of its FM algorithm implementation.
This article explains why Spark Streaming introduced backpressure, how the dynamic rate‑control mechanism works, and provides a detailed walkthrough of the relevant source code, including the RateController class, its registration, and the execution flow that adjusts ingestion rates to match processing capacity.
This article introduces Alink, Alibaba's open‑source machine‑learning platform built on Flink, detailing its core algorithms, performance advantages over Spark ML, version evolution, Maven and PyAlink installation steps, data‑source integrations, FM algorithm support, and unified file‑system operations for both batch and streaming workloads.
This article examines how Netflix migrated its video‑streaming platform to AWS, adopted a microservices architecture, and built a global CDN, detailing the system’s components, design goals such as high availability, low latency and scalability, and the trade‑offs and resilience techniques employed.
Flink SQL, the high‑level SQL interface for Apache Flink, offers language‑independent, dependency‑free, easy‑to‑use stream processing with advanced features such as DDL, UDFs, time semantics, windowing, pattern matching, and built‑in connectors, supporting data synchronization, batch‑stream fusion, Hive integration, and various product enhancements.
The article explains how Flink 1.11 deepens its integration with Hive, covering background, new connector features, simplified dependency management, enhanced Hive dialect, streaming writes and reads, temporal table joins, and how these capabilities enable a unified batch‑streaming data warehouse.
This guide walks you through using the Python httpx library to perform various HTTP methods—including GET, POST, PUT, DELETE, HEAD, and OPTIONS—handle query parameters, decode responses, work with JSON, custom headers, form data, file uploads, streaming, cookies, redirects, and authentication, all with clear code examples.
Lyft transformed its legacy data pipeline by designing a cloud‑native, Flink‑based near real‑time analytics platform that ingests billions of events, writes Parquet files to S3, leverages Presto for interactive queries, and implements multi‑stage non‑blocking ETL, fault‑tolerant back‑fill, and extensive performance optimizations.
This article details NetEase Cloud Music's evolution of a real‑time data warehouse built on Flink 1.9 and Calcite, covering platform scale, architectural design, metadata management, SDK simplifications, monitoring improvements, and concrete use cases such as AB‑testing, live reporting, and feature serving.
This article demonstrates how to develop a real-time TopN ranking feature in Apache Flink, covering stream setup, word count aggregation, global and grouped TopN calculations, and nested TopN strategies to mitigate hotspot issues, complete with Java code examples.
This article explains how Flink's Timer mechanism works, covering its usage in KeyedProcessFunction, the underlying TimerService and InternalTimerService implementations, the role of triggers, and the detailed code paths for processing‑time and event‑time timers, while highlighting performance considerations.
OPPO's data platform engineer Zhang Jun shares the design and implementation of OPPO's real‑time data warehouse built on Apache Flink, covering background, top‑level architecture, practical deployment, and future directions such as enhanced SQL development, resource scheduling, and automated configuration.
This article presents the concepts, architecture, and practical techniques of an online real‑time feature system used in intelligent risk‑control, covering feature definition, time‑window types, calculation functions, distributed processing, low‑latency storage, and operational challenges in high‑concurrency environments.
This article demonstrates how to use Flink 1.11's enhanced Hive integration to stream data from a Kafka source, write it into partitioned Hive tables with checkpoint‑driven commits, and read Hive tables as a continuous source using dynamic table options and table hints.
This article provides a comprehensive introduction to Kafka, covering its role as a messaging system, core concepts such as topics, partitions, producers, consumers, and messages, and then delves into the cluster architecture including replicas, consumer groups, controller coordination with Zookeeper, performance optimizations, log segmentation, and network design.
This article compares Apache Kafka and RabbitMQ, detailing their architectural differences, message handling models (pull vs push), performance characteristics, and ideal use cases, helping readers choose the appropriate messaging system for streaming, high‑throughput, or legacy protocol scenarios.
This article explains how to use Spark Streaming's Direct Approach with Kafka, manually manage offsets, and provides complete Java and Scala implementations—including a JavaKafkaManager class, a demo application, and a Scala KafkaManager—illustrating the creation of DirectKafkaInputDStream, offset handling, and integration with Spark.
This article presents a comprehensive guide to constructing real-time data warehouses using Apache Flink, covering the motivations, design principles, application scenarios, layer-by-layer architecture, metadata and lineage management, quality assurance, and the supporting toolchain for reliable streaming analytics.
This article examines Netflix’s cloud‑based microservices architecture, detailing its client, backend, CDN components, design goals such as high availability, low latency, scalability, and the trade‑offs, resilience mechanisms, and scalability strategies employed on AWS to support millions of global streaming users.
This article compares RabbitMQ and Apache Kafka, explaining their internal mechanisms, key differences in ordering, routing, timing, retention, fault tolerance, scalability, and consumer complexity, and provides guidance on when to choose each technology for modern micro‑service architectures.
This article surveys the rapidly evolving big data landscape by reviewing a wide range of Apache projects—including Hadoop, Spark, Flink, HBase, Kudu, Impala, Kafka, and others—detailing their core components, architectures, strengths, and typical use‑cases for building distributed data platforms.
This article compiles frequent Spark SQL, Spark Core, PySpark, and Streaming problems—such as filesystem errors, configuration pitfalls, memory limits, shuffle failures, and version incompatibilities—along with concise explanations of their causes and step‑by‑step remediation methods for big‑data environments.
The article explains how MQTT, a lightweight IoT messaging protocol, and Kafka, a distributed streaming platform, differ in architecture, purpose, and design goals despite both using publish/subscribe, and discusses their complementary integration via bridges such as EMQ X.
This article provides a comprehensive overview of Apache Kafka, explaining its distributed, fault‑tolerant architecture, horizontal scalability, disk‑based commit log, replication mechanisms, Streams API, KSQL, and why it is widely adopted as the backbone of event‑driven, high‑throughput systems.
This article explains how FunTouTiao builds a high‑performance real‑time analytics pipeline using Flink, Hive, and ClickHouse, covering business scenarios, hour‑level and second‑level Flink‑to‑Hive architectures, streaming file sink mechanics, multi‑user permissions, ClickHouse performance tricks, and future roadmap for unified stream‑batch storage.
This article details Shopee Singapore Data Team’s implementation of a Flink‑based real‑time data warehouse, covering background challenges, layered architecture integrating Kafka, HBase, Druid, Hive, streaming pipelines, job management, monitoring, and future plans to expand Flink SQL support.
This article summarizes the objectives, design principles, application scenarios, layer‑by‑layer construction methods, quality assurance mechanisms, and supporting tools for building a real‑time data warehouse using Apache Flink, providing practical guidance for data engineers and architects.
This article explains common scenarios where Kafka messages can be lost on the producer, consumer, or broker side, and provides practical configurations—including callbacks, acks, retries, manual offset commits, idempotent and transactional producers—to ensure reliable delivery and exactly‑once processing.
This article reviews common Kafka interview questions, covering delay queues, idempotence, replica states, offsets, message ordering, and handling duplicate consumption, and includes example code for enabling idempotent producers along with explanations of time‑wheel mechanisms and practical solutions to consumer rebalance issues.
QuTouTiao leverages Flink and ClickHouse to build a high‑performance real‑time analytics platform that supports hourly Hive pipelines and sub‑second ClickHouse queries, achieving sub‑second response for 80% of requests through streaming ingestion, exactly‑once semantics, multi‑cluster coordination, and optimized ClickHouse storage and connector designs.
This article explains how to compute the top‑N hot products or brands within a time window using Apache Flink, covering data modeling, event‑time handling, sliding windows, custom aggregation functions, and result sorting with complete Java code examples.
This article describes the evolution of a real‑time computing platform from SQL 1.0 built on Spark Structured Streaming to SQL 2.0 powered by Flink‑SQL, covering dynamic tables, continuous queries, dimension‑table joins, cache optimization, DDL extensions, platformization, operational challenges and future roadmap.
This article provides a detailed introduction to Spark Streaming, covering its architecture, DStream concepts, initialization, data sources, transformations, windowed aggregations, output operations, checkpointing, fault‑tolerance semantics, deployment, performance tuning, and monitoring for building reliable high‑throughput streaming applications.
Facing growing order volumes and strict timeliness demands, Cainiao’s tech team overhauled its real‑time data warehouse by redesigning data models, adopting Flink for streaming computation, upgrading data services, and exploring innovative tools, sharing practical lessons and future directions for large‑scale logistics analytics.
This article explains the MySQL communication protocol, result‑set structure, client library interfaces, and the difference between store‑result and streaming modes, then details how UPSQL Proxy 2.4.0 adopts streaming to reduce latency and memory usage in distributed database environments.
This comprehensive guide covers Kafka's definition, publish/subscribe model, key components, storage mechanisms, producer and consumer strategies, and reliability features such as ACK levels, ISR, and exactly‑once semantics, providing a solid foundation for real‑time big‑data processing.
This article shares Cainiao's practical experience in constructing a real-time data warehouse, covering the shortcomings of the previous architecture, the evolution of data models, the migration to Flink with advanced features like retraction and timer services, and the modernization of data services and tooling to support high‑throughput logistics scenarios.
This article explains the new unified TaskManager memory model introduced in Flink 1.10, detailing each memory component, its configuration parameters, how they map to JVM settings, and practical guidance for both standalone and containerized deployments, including a concrete YARN example.
This article outlines a comprehensive real‑time data system built on Apache Flink, covering its application scenarios, layered architecture, data model stratification, construction methods, and a concrete Flink SQL example for calculating UV metrics from Kafka‑sourced page‑view data.
This article presents a series of hands‑on big‑data exercises, including Spark Scala data analysis, Kafka topic creation and custom partitioning, and MySQL table design with Scala‑based streaming calculations, providing complete source code and step‑by‑step solutions for each task.
This article presents an in‑depth overview of Apache Flink's new OLAP engine, covering OLAP fundamentals, the three OLAP models, Flink's unified streaming‑batch‑OLAP architecture, performance optimizations, benchmark results, and future development directions.
This article explains the fundamental differences between RabbitMQ and Apache Kafka, covering asynchronous messaging patterns, the internal architectures of both systems, their respective strengths and weaknesses, and guidance on choosing the appropriate solution for various scenarios.
This article explains how Apache Flink's flexible state management can be leveraged to solve data correlation challenges in real‑time recommendation platforms, compares Flink with Spark and Storm, describes the underlying broadcast and managed state mechanisms, and provides a step‑by‑step implementation using Kafka, Druid, and custom broadcast functions.
This article reviews recent progress in Apache Flink's AI ecosystem, explaining how Flink unifies batch and stream processing for machine‑learning pipelines, introduces the Flink ML Pipeline and Alink library, describes the AI Flow framework for end‑to‑end ML workflows, and presents a novel mini‑batch streaming iteration mechanism to support both offline and online learning scenarios.
This article provides an in‑depth yet approachable overview of Kafka's core concepts—including its architecture, terminology, message‑sending pipeline, replication strategy, controller role, and consumer group rebalance mechanisms—helping readers quickly grasp how Kafka works as a high‑throughput distributed messaging and streaming platform.
This article explains how Apache Flink implements fault‑tolerant checkpointing using the Asynchronous Barrier Snapshot (ABS) algorithm, a localized version of the Chandy‑Lamport distributed snapshot, covering barriers, snapshot alignment, exactly‑once versus at‑least‑once semantics, and handling of cyclic dataflow graphs.
Tencent Cloud CKafka powers Tencent Classroom’s pandemic‑era online teaching by replacing a custom queue with a high‑performance, highly available, partition‑based message bus that scales to millions of real‑time interactions, offers configurable replication and tuning for reliability, and integrates with big‑data and streaming tools for analytics.
The talk details Cainiao’s evolution of its real‑time data warehouse architecture, covering the original 2016 model, compute and service challenges, the 2017 multi‑layer data model redesign, migration to Flink, practical cases of state retraction, timeout statistics, smart optimizations, and the unified data service platform.
The presentation by Bilibili's real‑time platform lead details the design and implementation of a Flink‑based streaming data platform, explains how AI workloads are integrated, shares architectural decisions and operational insights, and provides the full slide deck for knowledge dissemination.
Meituan-Dianping’s senior technical expert shares the evolution, architecture, and implementation of their Apache Flink‑based real‑time data warehouse platform, covering platform evolution, layered design, job and resource management, business warehouse use cases, and future development considerations.
This article details{32-64 words} OPPO's evolution from an offline data warehouse to a real‑time platform, describing the business scale, data‑mid platform architecture, migration strategy using Flink SQL, extensions like AthenaX, and practical use cases such as real‑time ETL, CTR calculation, and tag import.
This article explains how to handle dimension table joins in Apache Flink streaming by leveraging Async I/O to perform non‑blocking external lookups, provides detailed code examples for both synchronous and asynchronous functions, discusses configuration parameters, and outlines best practices and pitfalls.
The article outlines Tencent Classroom’s cloud VOD solution, detailing HLS streaming fundamentals, a Mongoose‑based local HTTP proxy with LFU caching and pre‑loading, performance optimizations for latency, buffering, security, and playback reliability, and common transcoding pitfalls with practical fixes, highlighting cloud migration benefits.
This article provides a comprehensive overview of Apache Spark Structured Streaming, describing its declarative API, the challenges of stream processing, the programming model with code examples, query planning, execution modes, production use cases, and performance benchmarks compared with other streaming systems.
The article explains how LocalServer pre‑caches video data to improve start‑up speed and playback smoothness, detailing when and how much data is cached, storage policies, handling of network interruptions, and practical testing points for developers.
The article provides a comprehensive overview of Apache Kafka 2.4.0, detailing its major new capabilities such as consumer replica fetching, progressive cooperative rebalancing, MirrorMaker 2.0, new Java authentication APIs, and extensive bug fixes, along with upgrade considerations and related resources.
The article explains Spark's default static resource allocation, analyzes the limitations of its Dynamic Resource Allocation (DRA) for streaming workloads, describes the internal Spark components and code paths involved, and proposes concrete design and configuration recommendations for implementing more responsive executor scaling.
Apache Kafka 2.4.0 introduces a range of new capabilities—including consumer replica fetching, incremental cooperative rebalancing, MirrorMaker 2.0, a new Java authorization API, KTable non‑key joins, administrative replica reassignment, protected REST endpoints, and offset deletion—along with numerous performance and stability improvements.
This article explains how to develop a real‑time TopN ranking feature using Apache Flink, covering both global and grouped TopN implementations, nested TopN strategies, and provides complete Java code snippets for environment setup, word counting, windowing, and custom TopN functions.
This article explains how to develop a real‑time ETL application using Apache Flink that reads events from Kafka, partitions them by event time into HDFS directories, and achieves exactly‑once processing through checkpointing, custom bucket assigners, and proper state backend configuration.
Alink, Alibaba’s open‑source machine‑learning platform built on Apache Flink, offers a rich library of batch and streaming algorithms, a Python API, iterative computation optimizations, and real‑world case studies, positioning it as a powerful AI solution for large‑scale, low‑latency data processing.
This article explains how to create custom Flink SourceFunction and SinkFunction implementations, demonstrates a RocketMQ source and an HBase sink with full code examples, and discusses checkpointing, event‑time handling, and deployment of the streaming job on a Flink‑on‑YARN cluster.
This article provides a comprehensive overview of Flink SQL window functions, detailing time‑based window types, their underlying implementation in the StreamExecGroupWindowAggregate operator, the processing flow of WindowOperator, timer handling, emit/trigger strategies, and practical code examples for Tumble, Hop, and Session windows.
The article demonstrates how to enrich Flink streaming ETL jobs with slowly changing dimension data by periodically loading MySQL tables into an in‑memory cache and performing a simple map‑side join within a custom RichMapFunction implementation.
This article provides a comprehensive overview of Apache Kafka, covering its fundamental capabilities, typical use cases, core components, key APIs, and essential concepts such as topics, partitions, segments, brokers, producers, and consumers, illustrated with diagrams.
This article presents a detailed overview of G7's AI‑plus‑Big‑Data‑plus‑IoT platform for logistics, describing its neutral open architecture, real‑time data pipelines using Kafka and Flink, Lambda‑style storage in HBase/Hive, and the resulting safety‑insurance and analytics capabilities.
This article explains the design goals, exactly‑once semantics, Java transaction API, internal components such as the transaction coordinator and log, data‑flow interactions, performance considerations, and practical tips for using Apache Kafka transactions in stream‑processing applications.
The CBVRP (Content-Based Video Relevance Prediction) challenge, co‑hosted by Hulu and ACM MM 2019, showcased the shift from user‑based collaborative filtering to content‑driven recommendation, highlighted winning teams and their papers, and underscored the ongoing research importance of cold‑start video recommendation for streaming platforms.
This article explains how to use Flink 1.9's built‑in Kafka connector, covering supported versions, Maven dependencies, consumer and producer configuration in Java and Scala, checkpointing, offset handling, partition discovery, timestamps, watermarks, and provides a complete runnable example.
This article examines Xiaomi's migration from Spark Streaming to Apache Flink, comparing scheduling strategies, mini‑batch versus true streaming, resource utilization, latency, and serialization mechanisms, and concludes with practical insights and custom optimization techniques for large‑scale data processing.
This article demonstrates how to build a one‑second‑refresh real‑time dashboard for e‑commerce order data using Apache Flink, Kafka, and Redis, covering JSON message parsing, processing‑time windows, stateful aggregation for site‑level KPIs, and efficient top‑N product ranking via Redis sorted sets.
This article details Beike's real‑time computing engine, covering its background, streaming platform built on Spark Streaming and Flink, data ingestion via Kafka, metadata handling, SQL‑based task development, monitoring, storage solutions, and future roadmap for resource management and AI‑enhanced monitoring.
qrpc is a lightweight, high‑performance RPC framework that adopts gRPC's streaming and bidirectional concepts without HTTP/2, offering a smaller binary, lower memory usage, up to three‑fold throughput gains, and flexible modes such as blocking, non‑blocking, streaming, push, and bidirectional calls, all demonstrated with Go code examples and real‑world use cases.
Debezium is an open‑source distributed platform built on Apache Kafka that turns existing databases into real‑time event streams by capturing row‑level changes via change data capture, offering source and embedded connectors, flexible topic routing, and features such as snapshots, filtering, masking, and monitoring.
This article explains the fundamentals of the two‑phase commit protocol, details its two stages (prepare and commit), discusses its advantages and drawbacks, and shows how Apache Flink implements 2PC for exactly‑once semantics with Kafka using the TwoPhaseCommitSinkFunction and related code examples.
