Tagged articles
Apache Kafka
41 articles · Page 1 of 1
This article analyzes Kafka's 2026 AI integration including the official MCP Server (KIP-1318) for natural language cluster management, Real-Time Context Engine for low-latency stream queries, Kafka Streams for agent session memory via KTables, A2A cross-platform agent collaboration, and three integration patterns with code examples, plus pros, cons, and use-case recommendations.
A2A ProtocolAI agentsApache Kafka
0 likes · 25 min read

This guide covers Kafka core concepts, two deployment approaches (traditional ZooKeeper-based and modern KRaft mode), detailed step-by-step installation commands for Linux and macOS, environment verification scripts, and a production-ready KRaft automation script with systemd integration.
Apache KafkaBash ScriptKRaft
0 likes · 36 min read

Apache Kafka 4.0 has been officially released as the first major version that runs entirely without Apache ZooKeeper, introducing KRaft mode, a new consumer group protocol (KIP‑848), early‑access queue support (KIP‑932), updated Java requirements, and other enhancements aimed at improving scalability, operability, and messaging versatility.
Apache KafkaKIP-848KIP-932
0 likes · 3 min read

Apache Kafka achieves high‑throughput, fault‑tolerant messaging by combining a partitioned log architecture with leader‑follower replication, asynchronous producer pipelines, configurable acknowledgments, page‑cache‑based sequential writes, zero‑copy transfers, batching, compression, and a multi‑reactor network model that together ensure scalability, reliability, and performance.
Apache Kafkareliabilitystreaming
0 likes · 30 min read

This article provides a comprehensive overview of Apache Kafka, covering its core features, architectural components, message flow, and common application scenarios such as log collection, decoupled messaging, activity tracking, operational monitoring, and stream processing.
Apache KafkaMessage Queuedistributed systems
0 likes · 6 min read

This article introduces Apache Kafka as a high‑throughput, scalable, fault‑tolerant distributed streaming platform, explains why it is chosen for real‑time data pipelines, and details key Kafka Streams concepts such as stream processing, interactive queries, stateful processing, windowing, serialization, and testing.
Apache KafkaKafka Streamsbig data
0 likes · 13 min read

This article details Agoda's evolution of Apache Kafka usage—from a two‑step logging architecture that separates developer concerns, through cluster layout, scaling metrics, monitoring and audit pipelines, to cost attribution, authentication, ACLs, and automation tools—highlighting trade‑offs and operational lessons learned.
Apache Kafkacost managementdata pipeline
0 likes · 17 min read

This article presents a detailed benchmark comparing Redpanda 23.1 and Apache Kafka 3.4.0 (with and without KRaft) across multiple AWS instance types, showing how Redpanda consistently delivers higher throughput and dramatically lower end‑to‑end latency, often outperforming Kafka by 4‑20× even with extra hardware.
Apache KafkaKRaftRedpanda
0 likes · 12 min read

This article explains the design, semantics, and practical usage of Apache Kafka's transaction API, covering why transactions are needed for exactly‑once processing, the underlying atomic multi‑partition writes, zombie fencing, consumer guarantees, Java API details, performance considerations, and operational best practices.
Apache KafkaJavaTransactions
0 likes · 19 min read

The Apache Kafka 3.3 release introduces production‑ready KRaft mode and a series of KIP enhancements across brokers, streams, and connect, detailing new metrics, API changes, and operational improvements while encouraging users to review the full release notes and try the updated platform.
Apache KafkaKIPKafka Connect
0 likes · 10 min read

The article explains how Kafka serves as the core of a real‑time data warehouse for search, detailing its advantages over traditional databases, integration with Flink for low‑latency stream processing, architectural patterns such as Lambda/Kappa, scaling challenges, and comprehensive monitoring using Kafka Eagle.
Apache KafkaFlinkSpark
0 likes · 15 min read

This article provides an in‑depth introduction to Apache Kafka, covering its distributed streaming platform fundamentals, message‑queue models, topic and partition design, broker and cluster roles, producer partitioning logic, reliability guarantees, consumer group assignors, offset management, and performance optimizations such as sequential disk writes and zero‑copy techniques.
Apache Kafkaconsumer groupsdistributed-streaming
0 likes · 25 min read

The article provides a comprehensive overview of Apache Kafka 3.0, detailing its core APIs, two main use‑cases, major feature additions, deprecations, KRaft consensus improvements, enhanced producer guarantees, and numerous KIP‑driven changes across the broker, client, Connect, Streams, and MirrorMaker components.
Apache KafkaKIPKRaft
0 likes · 14 min read

This article provides an in‑depth technical guide to Apache Kafka, covering its distributed streaming architecture, core concepts such as topics, partitions, brokers, producers and consumers, reliability guarantees, storage mechanisms, configuration parameters, and consumer assignment strategies, supplemented with Java code examples.
Apache KafkaConsumerPartitioning
0 likes · 24 min read


21CTOSep 27, 2021 · Big Data
A roundup of recent tech news covering China's crackdown on cryptocurrency, Huawei's upcoming programming language, the release of Apache Kafka 3.0, and other major developments in China's digital economy and industry leadership.
Apache KafkaHuaweiTechnology News
0 likes · 8 min read

Apache Kafka 3.0.0 introduces a host of enhancements—including deprecated Java 8/Scala 2.12 support, Raft metadata snapshots, stronger producer guarantees, MirrorMaker 2 upgrades, and Kafka Streams improvements—while continuing to serve real‑time data pipelines and streaming applications.
Apache KafkaKafka 3.0big data
0 likes · 3 min read

Apache Kafka 3.0.0 introduces numerous enhancements including deprecation of Java 8 and Scala 2.12 support, KRaft metadata snapshots, stronger default producer delivery guarantees, expanded Connect and Streams APIs, updated MirrorMaker 2 configuration, and many KIP-driven feature and API changes for improved streaming and event processing.
Apache KafkaEvent ProcessingKIP
0 likes · 15 min read

Apache Kafka 2.8.0 introduces several significant enhancements, including a new group API, mutual TLS authentication for SASL_SSL listeners, JSON request/response logging, broker connection rate limiting, topic identifiers, self‑managed quorum replacing ZooKeeper, and numerous improvements to Streams and Connect APIs for more reliable real‑time data pipelines.
Apache KafkaKafka 2.8.0big data
0 likes · 2 min read

Kafka 2.8 replaces ZooKeeper with an internal Quorum controller (KRaft), allowing users to run Kafka without external coordination, reducing resource usage, improving performance, and supporting larger clusters, while noting that some features like ACLs and partition reassignment are still pending.
Apache KafkaKRaftZooKeeper
0 likes · 4 min read

Apache Kafka is a distributed, fault‑tolerant streaming platform that uses broker clusters, topic partitions, and replicated logs to provide publish/subscribe and queue messaging, configurable retention, strong ordering within partitions, producer acknowledgments, consumer groups, and performance optimizations such as batching and zero‑copy for real‑time data pipelines.
Apache KafkaConsumerMessage Queue
0 likes · 26 min read

This article explains the purpose, semantics, and design of Apache Kafka’s transaction API, detailing how it enables exactly‑once processing for stream‑processing applications, the role of transaction coordinators and logs, Java API usage, performance considerations, and best‑practice guidance.
Apache KafkaJava APITransactions
0 likes · 19 min read

This article explains how to integrate Apache Kafka with Spring Boot, covering error recovery with SeekToCurrentErrorHandler, deserialization handling, domain‑object conversion, multiple listener routing, and transaction support, while providing complete code examples for each feature.
Apache KafkaMessage ConversionTransactions
0 likes · 10 min read

Apache Kafka 2.3 introduced static members and incremental cooperative rebalancing to reduce disruptive global rebalances, allowing workers to retain assignments during failures, schedule delayed rebalances, and improve scalability for Kafka Connect clusters, balancing availability and fault tolerance.
Apache KafkaIncremental RebalancingKafka Connect
0 likes · 12 min read

This article introduces KSQL as a lightweight streaming SQL engine for Apache Kafka, explains its architecture and core concepts of streams and tables, and provides step‑by‑step deployment instructions, command‑line examples for creating streams/tables, querying data, and managing persistent queries.
Apache KafkaKSQLKafka
0 likes · 10 min read

The article provides a comprehensive overview of Apache Kafka 2.4.0, detailing its major new capabilities such as consumer replica fetching, progressive cooperative rebalancing, MirrorMaker 2.0, new Java authentication APIs, and extensive bug fixes, along with upgrade considerations and related resources.
Apache KafkaVersion 2.4.0big data
0 likes · 26 min read

Apache Kafka 2.4.0 introduces a range of new capabilities—including consumer replica fetching, incremental cooperative rebalancing, MirrorMaker 2.0, a new Java authorization API, KTable non‑key joins, administrative replica reassignment, protected REST endpoints, and offset deletion—along with numerous performance and stability improvements.
Apache KafkaKStreamsKafka 2.4.0
0 likes · 3 min read

This article explains how Spring Cloud Data Flow, together with Spring Cloud Skipper, enables developers to design, deploy, and manage event‑stream pipelines on Apache Kafka, covering ecosystem overview, pipeline components, Docker‑based local setup, stream creation, debugging, monitoring, and integration of Kafka Streams applications.
Apache KafkaKubernetesSpring Cloud Data Flow
0 likes · 15 min read

This article explains how to integrate Apache Kafka with Spring Boot, covering error handling with SeekToCurrentErrorHandler, deserialization error handling, type‑inferred message conversion, multi‑listener routing, and transactional message processing, providing code examples and configuration details for each feature.
Apache KafkaJavaMessage Conversion
0 likes · 10 min read

This article explores Apache Kafka’s evolution, core messaging and stream processing capabilities, typical use cases, internal storage mechanisms, API choices, and best practices for deploying Kafka on Kubernetes, providing readers with comprehensive guidance to assess suitability and implement effective Kafka solutions.
Apache KafkaKafka APIsKubernetes
0 likes · 16 min read

This article explains how Apache Flink’s TwoPhaseCommitSinkFunction, introduced in version 1.4, enables end-to-end exactly-once semantics when integrated with Apache Kafka, detailing the checkpoint mechanism and the two-phase commit protocol that ensures reliable data processing.
Apache FlinkApache KafkaTwo-Phase Commit
0 likes · 4 min read

This article explains Apache Kafka as a distributed streaming platform, detailing its key features, core APIs, topic and log architecture, partitioning, consumer groups, guarantees, and how it serves both messaging and storage roles for real‑time and batch processing in big‑data environments.
Apache Kafkadistributed systemsmessage queues
0 likes · 13 min read

This article presents New Relic’s 20 best‑practice recommendations for Apache Kafka, covering partitions, consumers, producers, and brokers, to help engineers design, configure, and monitor high‑throughput, reliable streaming pipelines at scale.
Apache Kafkahigh-throughput
0 likes · 13 min read

This article presents 20 practical best‑practice recommendations—from partition sizing and consumer tuning to producer configuration and broker optimization—to help engineers build scalable, reliable, and high‑throughput Apache Kafka clusters for large‑scale applications.
Apache KafkaBrokersConsumers
0 likes · 16 min read

This article provides a detailed comparison between Apache Pulsar and Apache Kafka, covering their message consumption models (queue vs. stream), subscription types, acknowledgment mechanisms, retention policies, and underlying layered architecture, highlighting Pulsar's unified API and segment‑based storage advantages.
Apache KafkaApache PulsarMessage Model
0 likes · 21 min read


21CTONov 7, 2018 · Big Data
Data streams, akin to endless rivers, enable continuous, real-time processing of diverse sources such as IoT telemetry, web logs, and e-commerce events, offering advantages over batch processing, while presenting challenges like scalability and fault tolerance, and are supported by tools like Kinesis, Kafka, Flink, and Storm.
Amazon KinesisApache KafkaData Streams
0 likes · 6 min read


21CTOJul 11, 2018 · Backend Development
This article explains what data streams are, how they differ from arrays, the types of streams in Node.js, demonstrates Java Stream operations, and introduces popular streaming platforms like Apache Kafka and Amazon Kinesis, highlighting their core features and real‑time processing capabilities.
Amazon KinesisApache KafkaData Streams
0 likes · 7 min read

This article explains Apache Kafka’s evolution and adoption by Fortune‑500 firms, outlines its two core capabilities—messaging (queue and publish/subscribe) and stream processing via the Java Stream API—provides example code, typical use cases, and guidance on scenarios where Kafka may not be the best solution.
Apache Kafkastream processinguse cases
0 likes · 5 min read

This article explains Apache Kafka’s distributed publish‑subscribe design, its core components, storage model, broker behavior, integration with ZooKeeper, performance comparisons with RabbitMQ and ActiveMQ, and provides a practical example application illustrating producer and consumer APIs.
Apache Kafkabackend developmentmessage queues
0 likes · 17 min read

The article explains how Apache Kafka has become a pivotal, high‑scalable publish‑subscribe system in the big‑data ecosystem, addressing the limitations of traditional databases, enabling real‑time data integration across specialized distributed systems, and shaping future data‑governance practices.
Apache Kafkadata integrationstreaming
0 likes · 7 min read


21CTOSep 14, 2015 · Backend Development
This article explains Apache Kafka's distributed publish‑subscribe design, core components, storage model, ZooKeeper coordination, performance benchmarks against ActiveMQ and RabbitMQ, and provides Java producer and consumer code examples for building high‑throughput messaging applications.
Apache KafkaDistributed MessagingMessage Queue
0 likes · 16 min read

The article explains the fundamentals of stream processing, event sourcing, and complex event processing, comparing raw event storage with aggregated results, illustrating architectures with Kafka, Samza, and other frameworks, and highlighting benefits such as scalability, flexibility, and decoupling for modern data‑driven systems.
Apache KafkaApache Samzabig data
0 likes · 11 min read
