BestHub
Sign in
Tagged articles
65 articles
Page 1 of 1
Architect Chen
Architect Chen
Mar 3, 2026 · Backend Development

Preventing Kafka Duplicate Consumption with Idempotent Design

This article explains practical strategies to avoid duplicate message consumption in Kafka, covering business idempotency with unique IDs, database or Redis deduplication tables, enabling producer idempotence, consumer-side checks, and Kafka's transaction-based exactly‑once semantics, along with their trade‑offs and suitable scenarios.

BackendExactly-OnceIdempotency
0 likes · 4 min read
Preventing Kafka Duplicate Consumption with Idempotent Design
Ray's Galactic Tech
Ray's Galactic Tech
Jan 12, 2026 · Backend Development

How to Achieve Exactly‑Once Processing with RabbitMQ: 6 Practical Strategies

This article presents a comprehensive, end‑to‑end solution for preventing duplicate consumption in RabbitMQ by combining architecture design, routing strategies, idempotent controls, state machines, and ACK mechanisms, and details six concrete implementation options with code examples.

Exactly-OnceIdempotencyMessage Deduplication
0 likes · 9 min read
How to Achieve Exactly‑Once Processing with RabbitMQ: 6 Practical Strategies
Architect Chen
Architect Chen
Jan 2, 2026 · Backend Development

Preventing Duplicate Consumption in Kafka: Design, Idempotence, and Configuration Strategies

This guide explains how to avoid duplicate message consumption in Kafka by designing unique identifiers, implementing consumer-side idempotence with deduplication tables, leveraging Kafka’s transactional features, and establishing system-level safeguards and monitoring to ensure reliable, exactly‑once processing.

Duplicate ConsumptionExactly-OnceIdempotence
0 likes · 4 min read
Preventing Duplicate Consumption in Kafka: Design, Idempotence, and Configuration Strategies
Ray's Galactic Tech
Ray's Galactic Tech
Oct 18, 2025 · Backend Development

How to Implement Exactly-Once Transactions in Spring Boot with Kafka

This guide explains how to configure Spring Boot and Kafka to achieve Exactly-Once semantics, covering core concepts, Maven dependencies, YAML settings, sample code for producers and consumers, execution flow, and advanced tips for reliable transactional messaging.

BackendExactly-Oncespring-boot
0 likes · 8 min read
How to Implement Exactly-Once Transactions in Spring Boot with Kafka
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jul 9, 2025 · Backend Development

How to Prevent Duplicate Consumption in Kafka: Practical Strategies

This article explains why Kafka’s at‑least‑once delivery can cause duplicate message processing, outlines the business risks of such duplicates, and presents four practical solutions—including idempotent design, manual offset commits, exactly‑once semantics, and dead‑letter queues—to ensure reliable consumption.

Dead Letter QueueDuplicate ConsumptionExactly-Once
0 likes · 6 min read
How to Prevent Duplicate Consumption in Kafka: Practical Strategies
macrozheng
macrozheng
Apr 11, 2025 · Backend Development

How to Guarantee Exactly‑Once Message Consumption in High‑Concurrency Systems

This article explains common causes of duplicate message consumption in high‑traffic systems and presents a three‑layer defense—producer idempotence, broker de‑duplication, and consumer idempotent design—plus monitoring and reconciliation strategies to achieve reliable exactly‑once processing.

Exactly-OnceIdempotenceKafka
0 likes · 7 min read
How to Guarantee Exactly‑Once Message Consumption in High‑Concurrency Systems
StarRocks
StarRocks
Jan 30, 2024 · Big Data

How InLong Guarantees Exactly‑Once Real‑Time Writes to StarRocks

This article explains how Apache InLong provides automatic, secure, high‑performance real‑time data transfer to StarRocks, detailing the transactional Stream Load API, the two‑phase commit process, Flink‑based ingestion architecture, exactly‑once guarantees, and performance test results across different parallelism levels.

Big DataExactly-OnceInLong
0 likes · 11 min read
How InLong Guarantees Exactly‑Once Real‑Time Writes to StarRocks
JD Tech
JD Tech
Jun 16, 2023 · Big Data

Comprehensive Introduction to Apache Kafka: Architecture, Features, and Best Practices

This article provides a detailed overview of Apache Kafka, covering its distributed streaming architecture, storage mechanisms, replication, consumer groups, compression techniques, exactly‑once semantics, configuration tips, and performance optimizations for building reliable high‑throughput data pipelines.

Big DataDistributed StreamingExactly-Once
0 likes · 19 min read
Comprehensive Introduction to Apache Kafka: Architecture, Features, and Best Practices
Architects Research Society
Architects Research Society
Jun 4, 2023 · Big Data

Understanding Transactions in Apache Kafka

This article explains the design, semantics, and practical usage of Apache Kafka's transaction API, covering why transactions are needed for exactly‑once processing, the underlying atomic multi‑partition writes, zombie fencing, consumer guarantees, Java API details, performance considerations, and operational best practices.

Apache KafkaDistributed SystemsExactly-Once
0 likes · 19 min read
Understanding Transactions in Apache Kafka
Architects Research Society
Architects Research Society
Apr 25, 2023 · Big Data

Understanding Transactions in Apache Kafka: Semantics, API, and Practical Guidance

This article explains the purpose, semantics, and design of Apache Kafka's transaction API, describes how exactly‑once processing is achieved in stream‑processing applications, outlines the Java client usage, and discusses the internal components, performance considerations, and best‑practice tips for developers.

Distributed SystemsExactly-OnceKafka
0 likes · 16 min read
Understanding Transactions in Apache Kafka: Semantics, API, and Practical Guidance
Architects Research Society
Architects Research Society
Mar 15, 2023 · Big Data

Understanding Transactions in Apache Kafka: Semantics, API, and Practical Considerations

This article explains why exactly‑once semantics are needed for stream‑processing applications, describes Kafka's transactional model and semantics, details the Java transaction API and its usage, and discusses the internal components, performance trade‑offs, and practical guidelines for building reliable Kafka‑based pipelines.

Distributed SystemsExactly-OnceKafka
0 likes · 17 min read
Understanding Transactions in Apache Kafka: Semantics, API, and Practical Considerations
Top Architect
Top Architect
Aug 28, 2022 · Big Data

Understanding Kafka Architecture: Topics, Partitions, Producers, Consumers, Offsets, and Transactions

Kafka is a high‑performance distributed message queue that uses topics, partitions, and replicas for scalability, with producers writing messages, consumers reading them via consumer groups, and advanced features such as offset management, delivery semantics, rebalancing, and exactly‑once transactional processing, all coordinated by Zookeeper.

ConsumersDistributed MessagingExactly-Once
0 likes · 17 min read
Understanding Kafka Architecture: Topics, Partitions, Producers, Consumers, Offsets, and Transactions
Alibaba Cloud Native
Alibaba Cloud Native
Jun 25, 2022 · Cloud Native

How Serverless Function Compute Guarantees Exactly‑Once Task Execution

This article explains the technical details of how a serverless function compute platform provides exactly‑once task triggering semantics, covering message delivery models, system‑side failover handling, user‑side deduplication with TaskID, and a Go SDK example for reliable asynchronous job execution.

Exactly-Oncecloud-nativetask deduplication
0 likes · 10 min read
How Serverless Function Compute Guarantees Exactly‑Once Task Execution
Alibaba Cloud Developer
Alibaba Cloud Developer
Jun 13, 2022 · Cloud Computing

How Serverless Functions Achieve Exactly‑Once Task Deduplication

Serverless Task provides exactly‑once task triggering by combining system‑side failover protection with user‑level deduplication, using a robust queue, scheduler, and globally unique TaskID to ensure each task is executed only once even under failures, network partitions, or retries.

Exactly-OnceFunction Computecloud computing
0 likes · 10 min read
How Serverless Functions Achieve Exactly‑Once Task Deduplication
Alibaba Cloud Native
Alibaba Cloud Native
Feb 22, 2022 · Big Data

Why RocketMQ-Streams Delivers High‑Performance, Low‑Resource Stream Computing

RocketMQ-Streams targets massive data, high‑filtering, lightweight windowed computations with a lightweight, high‑performance design that runs on as little as 1 CPU core and 1 GB RAM, offering 2‑5× speed gains over traditional big‑data engines and supporting Flink‑compatible SQL, UDFs, and cloud‑native deployment.

Exactly-OnceFlink SQLRocketMQ-Streams
0 likes · 10 min read
Why RocketMQ-Streams Delivers High‑Performance, Low‑Resource Stream Computing
Code Ape Tech Column
Code Ape Tech Column
Jan 21, 2022 · Backend Development

Message Deduplication and Exactly-Once Semantics in RocketMQ: Strategies and Implementation

This article explains the challenges of at‑least‑once delivery in distributed message middleware like RocketMQ, examines simple and concurrent deduplication techniques, and presents both transaction‑based and non‑transactional exactly‑once solutions using database tables or Redis, along with practical Java code examples.

Exactly-OnceRocketMQdeduplication
0 likes · 17 min read
Message Deduplication and Exactly-Once Semantics in RocketMQ: Strategies and Implementation
Top Architect
Top Architect
Oct 24, 2021 · Backend Development

Message Deduplication and Exactly‑Once Semantics in RocketMQ

This article explains why RocketMQ guarantees at‑least‑once delivery, describes the three typical duplicate‑message scenarios, compares transaction‑based and non‑transactional deduplication approaches (including a Redis‑based solution), provides sample SQL and Java code, and discusses the limitations and best‑practice recommendations for achieving idempotent message consumption.

Exactly-OnceIdempotencyMessage Deduplication
0 likes · 20 min read
Message Deduplication and Exactly‑Once Semantics in RocketMQ
Alibaba Cloud Developer
Alibaba Cloud Developer
Oct 13, 2021 · Big Data

Why “Exactly‑Once” Doesn’t Guarantee Consistency in Stream Processing

This article examines the true meaning of consistency in stream computing, clarifies common misconceptions about exactly‑once semantics, formalizes consistency challenges, and reviews how major stream engines such as Google MillWheel, Apache Flink, Kafka Streams, and Spark Streaming implement end‑to‑end consistency.

Big DataExactly-Oncefault tolerance
0 likes · 29 min read
Why “Exactly‑Once” Doesn’t Guarantee Consistency in Stream Processing
Java Interview Crash Guide
Java Interview Crash Guide
Oct 13, 2021 · Backend Development

How to Achieve Exactly-Once Message Processing with RocketMQ Deduplication

Message middleware guarantees at-least-once delivery, but duplicate deliveries can cause issues; this article explains RocketMQ’s three duplication scenarios, explores simple and advanced deduplication strategies—including database-transaction and non-transactional approaches using Redis—and provides practical code samples for implementing reliable exactly-once processing.

Distributed SystemsExactly-OnceMessage Queue
0 likes · 21 min read
How to Achieve Exactly-Once Message Processing with RocketMQ Deduplication
Java Backend Technology
Java Backend Technology
Sep 20, 2021 · Backend Development

How to Achieve Exactly‑Once Message Processing in RocketMQ Without Transactions

This article explains why message middleware guarantees at‑least‑once delivery, the problems caused by duplicate consumption, and presents both transaction‑based and non‑transactional deduplication solutions—including a Redis‑backed idempotent table—to achieve exactly‑once semantics in RocketMQ.

Distributed SystemsExactly-OnceIdempotence
0 likes · 19 min read
How to Achieve Exactly‑Once Message Processing in RocketMQ Without Transactions
Java Architect Essentials
Java Architect Essentials
Sep 17, 2021 · Backend Development

How to Achieve Exactly‑Once Message Processing in RocketMQ Without Transactions

This article analyzes the at‑least‑once guarantee of message middleware, explains why duplicate deliveries occur, compares simple database‑based deduplication methods, explores concurrency challenges, and presents a non‑transactional, status‑driven idempotence solution using MySQL or Redis with practical code examples and limitations.

Distributed SystemsExactly-OnceIdempotence
0 likes · 19 min read
How to Achieve Exactly‑Once Message Processing in RocketMQ Without Transactions
Architecture Digest
Architecture Digest
Sep 17, 2021 · Backend Development

Message Deduplication and Exactly-Once Semantics in RocketMQ

This article explains why message middleware guarantees at‑least‑once delivery, describes three common duplication scenarios in RocketMQ, and presents both transactional and non‑transactional deduplication solutions—including SQL examples and a Redis‑based idempotence library—to achieve exactly‑once processing.

Distributed SystemsExactly-OnceIdempotence
0 likes · 19 min read
Message Deduplication and Exactly-Once Semantics in RocketMQ
Tencent Cloud Developer
Tencent Cloud Developer
Sep 14, 2021 · Backend Development

A Comparative Overview of Transactional Messaging in RocketMQ, Kafka, and Pulsar

The article compares how RocketMQ, Apache Kafka, and Apache Pulsar implement transactional messaging—each using a two‑phase commit with half‑messages or transaction buffers, distinct coordinators, and idempotent producers—to provide atomicity and either strong consistency (RocketMQ) or exactly‑once delivery for high‑throughput stream processing (Kafka and Pulsar).

Distributed SystemsExactly-OnceKafka
0 likes · 22 min read
A Comparative Overview of Transactional Messaging in RocketMQ, Kafka, and Pulsar
Java Interview Crash Guide
Java Interview Crash Guide
Aug 30, 2021 · Backend Development

How to Achieve Exactly-Once Message Processing with RocketMQ Deduplication

Message middleware guarantees at-least-once delivery, but repeated deliveries cause duplicate processing; this article explains RocketMQ's duplicate scenarios, explores simple and advanced deduplication techniques—including database-transactional and Redis-based idempotent tables—and provides practical Java code for implementing exactly-once semantics.

Distributed SystemsExactly-OnceIdempotence
0 likes · 20 min read
How to Achieve Exactly-Once Message Processing with RocketMQ Deduplication
Programmer DD
Programmer DD
Jul 22, 2021 · Backend Development

How to Achieve Exactly‑Once Message Processing in RocketMQ Without Heavy Transactions

This article explains why message middleware guarantees at‑least‑once delivery, the challenges of duplicate consumption, and presents both simple and advanced deduplication strategies—including transactional and non‑transactional approaches using relational databases or Redis—to achieve effectively exactly‑once semantics in RocketMQ.

Exactly-OnceIdempotenceMessage Deduplication
0 likes · 18 min read
How to Achieve Exactly‑Once Message Processing in RocketMQ Without Heavy Transactions
Selected Java Interview Questions
Selected Java Interview Questions
Jul 19, 2021 · Backend Development

Message Queue Concepts and Consumption Scenarios

This article explains the core concepts of message queues—including producers, consumers, messages, brokers, and push/pull delivery—and analyzes three consumption scenarios: at‑most‑once, at‑least‑once, and exactly‑once, detailing the required producer, broker, and consumer behaviors for each.

ConsumerExactly-OnceMessage Queue
0 likes · 6 min read
Message Queue Concepts and Consumption Scenarios
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 9, 2021 · Big Data

Understanding Kafka: Use Cases, Reliability, Storage, Replication, Consumer Assignment, Transactions, and Exactly-Once Semantics

This article explains why Kafka is used, its buffering, decoupling, redundancy and robustness benefits, details the ack reliability levels, storage design, replica synchronization, ISR handling, consumer partition assignment strategies, transaction support, exactly‑once semantics, and why read‑write separation is not provided.

Consumer AssignmentExactly-OnceMessage Queue
0 likes · 20 min read
Understanding Kafka: Use Cases, Reliability, Storage, Replication, Consumer Assignment, Transactions, and Exactly-Once Semantics
Tencent Cloud Middleware
Tencent Cloud Middleware
Jun 30, 2021 · Fundamentals

Understanding Apache Pulsar Transactions: Core Concepts and Workflow

Apache Pulsar 2.8.0 introduces transaction support, featuring a Transaction Coordinator, Transaction Buffer, Transaction Log, Transaction ID, and Pending Acknowledge State, with a detailed workflow that ensures exactly‑once semantics for stream processing, contrasting its design with Kafka’s approach.

Apache PulsarExactly-OnceKafka Comparison
0 likes · 13 min read
Understanding Apache Pulsar Transactions: Core Concepts and Workflow
Spring Full-Stack Practical Cases
Spring Full-Stack Practical Cases
Apr 12, 2021 · Big Data

Master Kafka’s Delivery Guarantees: At‑Most, At‑Least, and Exactly‑Once Explained

This article explains Kafka’s three delivery semantics—At most once, At least once, and Exactly once—from both producer and consumer perspectives, details the required configuration settings, and discusses how Kafka ensures idempotence, transaction support, and prevents data loss, duplication, and ordering issues.

Delivery GuaranteesExactly-OnceIdempotence
0 likes · 23 min read
Master Kafka’s Delivery Guarantees: At‑Most, At‑Least, and Exactly‑Once Explained
Architect
Architect
Feb 14, 2021 · Backend Development

Message Idempotency and Exactly‑Once Processing in RocketMQ

This article explains why message middleware like RocketMQ guarantees at‑least‑once delivery, the resulting duplicate‑delivery problem, and presents both transaction‑based and non‑transactional idempotency solutions—including select‑for‑update, optimistic locking, and a Redis‑backed deduplication table—to achieve exactly‑once semantics in distributed systems.

Distributed SystemsExactly-OnceRocketMQ
0 likes · 16 min read
Message Idempotency and Exactly‑Once Processing in RocketMQ
Architect
Architect
Sep 17, 2020 · Big Data

Kafka Exactly-Once Semantics and Transaction API Overview

This article explains Kafka's exactly‑once semantics and transaction support, detailing the new producer API methods, related exceptions, configuration parameters, and a sample application illustrating how to initialize, begin, process, and commit or abort transactions while ensuring idempotent and atomic message handling.

ConfigurationExactly-OnceIdempotence
0 likes · 19 min read
Kafka Exactly-Once Semantics and Transaction API Overview
Architecture Digest
Architecture Digest
Jun 24, 2020 · Big Data

Preventing Message Loss and Achieving Exactly‑Once Semantics in Kafka

This article explains common scenarios where Kafka messages can be lost on the producer, consumer, or broker side, and provides practical configurations—including callbacks, acks, retries, manual offset commits, idempotent and transactional producers—to ensure reliable delivery and exactly‑once processing.

Exactly-OnceIdempotenceMessage Loss
0 likes · 10 min read
Preventing Message Loss and Achieving Exactly‑Once Semantics in Kafka
Youzan Coder
Youzan Coder
Feb 28, 2020 · Big Data

Flink Checkpoint Principle Analysis and Failure Cause Investigation

The article thoroughly explains Apache Flink’s checkpoint mechanism—including state types, coordinator workflow, exactly‑once versus at‑least‑once semantics, common failure sources such as code exceptions, storage or network issues, and practical configuration tips like interval settings, local recovery and externalized checkpoints.

Apache FlinkCheckpointExactly-Once
0 likes · 15 min read
Flink Checkpoint Principle Analysis and Failure Cause Investigation
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 9, 2019 · Big Data

Building a Real‑Time ETL Pipeline with Apache Flink: Kafka to HDFS with Exactly‑Once Guarantees

This article explains how to develop a real‑time ETL application using Apache Flink that reads events from Kafka, partitions them by event time into HDFS directories, and achieves exactly‑once processing through checkpointing, custom bucket assigners, and proper state backend configuration.

Apache FlinkBig DataExactly-Once
0 likes · 11 min read
Building a Real‑Time ETL Pipeline with Apache Flink: Kafka to HDFS with Exactly‑Once Guarantees
58 Tech
58 Tech
Oct 10, 2019 · Big Data

Optimizing Real‑Time Feature Extraction at 58.com: Migrating from Spark Streaming to Flink

This article describes how 58.com’s commercial engineering team redesigned its real‑time feature‑mining pipeline—replacing a minute‑level Spark Streaming framework with Flink—to achieve sub‑second latency, higher throughput, stronger fault‑tolerance, and end‑to‑end exactly‑once semantics for user‑profile generation in the second‑hand‑car recommendation scenario.

Big DataExactly-OnceFlink
0 likes · 14 min read
Optimizing Real‑Time Feature Extraction at 58.com: Migrating from Spark Streaming to Flink
dbaplus Community
dbaplus Community
Sep 10, 2019 · Big Data

Why Exactly‑Once Processing Is So Hard in Distributed Systems (And How to Tackle It)

This article explores the two toughest problems in distributed stream processing—exactly‑once message handling and ordering—by dissecting the underlying impossibility of perfect failure detectors, the liveness‑vs‑safety trade‑off, zombie processes, and the practical solutions employed by systems such as Flink, Kafka Streams, MillWheel, and Spark.

ConsensusDistributed SystemsExactly-Once
0 likes · 81 min read
Why Exactly‑Once Processing Is So Hard in Distributed Systems (And How to Tackle It)
Architecture Digest
Architecture Digest
May 5, 2019 · Big Data

Kafka Architecture Overview: Topics, Partitions, Producers, Consumers, Replication, Leader Election, Offsets, Rebalance, Delivery Semantics, and Transactions

This article provides a comprehensive overview of Kafka's architecture, covering topics, partitions, producer and consumer workflows, replication and leader election, offset management, consumer group coordination, rebalance processes, delivery semantics (at‑most‑once, at‑least‑once, exactly‑once), transactional messaging, and underlying file and configuration details.

Distributed MessagingExactly-OnceKafka
0 likes · 16 min read
Kafka Architecture Overview: Topics, Partitions, Producers, Consumers, Replication, Leader Election, Offsets, Rebalance, Delivery Semantics, and Transactions
Big Data Technology Architecture
Big Data Technology Architecture
Apr 22, 2019 · Big Data

Comparison of Apache Spark and Apache Flink: Programming Models, Streaming, State Management, and Exactly-Once Semantics

This article compares Apache Spark and Apache Flink, outlining their programming models, streaming mechanisms, state management, time semantics, and exactly‑once guarantees, and highlights the strengths and differences of each framework for batch and real‑time big‑data processing.

Apache FlinkApache SparkExactly-Once
0 likes · 8 min read
Comparison of Apache Spark and Apache Flink: Programming Models, Streaming, State Management, and Exactly-Once Semantics
ITPUB
ITPUB
Mar 28, 2019 · Big Data

Why Pravega Matters: Native Stream Storage for Low‑Latency, Exactly‑Once Data Pipelines

Pravega, Dell’s native stream storage project, addresses the challenges of modern low‑latency, exactly‑once stream processing by combining tiered storage, Apache BookKeeper, and seamless Flink integration, offering a unified solution that reduces development, storage, and operational costs compared to traditional message systems like Kafka.

Apache FlinkExactly-OnceKafka Comparison
0 likes · 10 min read
Why Pravega Matters: Native Stream Storage for Low‑Latency, Exactly‑Once Data Pipelines
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 13, 2019 · Big Data

Understanding Fault Tolerance and Exactly-Once Semantics in Apache Flink

This article explains Apache Flink's fault‑tolerance mechanisms, including checkpointing, barrier alignment, the differences between At‑Least‑Once and Exactly‑Once semantics, configuration options, incremental checkpointing, and the requirements for external sources and sinks to achieve end‑to‑end exactly‑once processing.

Apache FlinkBig DataExactly-Once
0 likes · 15 min read
Understanding Fault Tolerance and Exactly-Once Semantics in Apache Flink
NetEase Game Operations Platform
NetEase Game Operations Platform
Jan 25, 2019 · Big Data

Understanding Exactly-Once Semantics in Apache Flink: Challenges and Implementation

This article analyzes the difficulties of achieving exactly-once delivery in Apache Flink, explains the distinction between state and end‑to‑end semantics, and details how idempotent and transactional sinks—illustrated with the Bucketing File Sink—realize exactly‑once guarantees through checkpoint‑based two‑phase commit.

Big DataExactly-OnceFlink
0 likes · 13 min read
Understanding Exactly-Once Semantics in Apache Flink: Challenges and Implementation
21CTO
21CTO
Sep 5, 2018 · Fundamentals

Why Exactly‑Once Delivery Is Impossible: Understanding Message Delivery Semantics

The article explains how messages travel in monolithic and distributed systems, why network communication is inherently unreliable, and how different delivery semantics—at‑most‑once, at‑least‑once, and exactly‑once—affect reliability, ordering, and protocol choices such as AMQP and MQTT.

At-Least-OnceDistributed SystemsExactly-Once
0 likes · 14 min read
Why Exactly‑Once Delivery Is Impossible: Understanding Message Delivery Semantics
Architecture Digest
Architecture Digest
Aug 7, 2018 · Big Data

Apache Kafka Overview, Architecture, and Sample Producer/Consumer Code

This article provides a comprehensive overview of Apache Kafka, comparing it with ActiveMQ, explaining its distributed architecture, topics, partitions, consumption models, high‑availability mechanisms, exactly‑once semantics, and includes detailed Java producer and consumer code examples for practical implementation.

Big DataConsumerDistributed Messaging
0 likes · 22 min read
Apache Kafka Overview, Architecture, and Sample Producer/Consumer Code
Meitu Technology
Meitu Technology
Aug 2, 2018 · Big Data

Spark Streaming vs Flink – Architecture, Scheduling & Fault Tolerance

This article compares Spark Streaming and Flink across runtime models, component roles, programming APIs, task scheduling, time semantics, dynamic Kafka partition detection, fault‑tolerance mechanisms, exactly‑once guarantees, and back‑pressure handling, providing code examples and practical insights for real‑time data processing.

Dynamic Partition DetectionExactly-OnceFlink
0 likes · 23 min read
Spark Streaming vs Flink – Architecture, Scheduling & Fault Tolerance
Beike Product & Technology
Beike Product & Technology
Mar 9, 2018 · Big Data

How Lianjia Built a Low‑Latency Real‑Time Data Platform with Spark Streaming

This article details Lianjia's journey of designing and implementing a low‑latency, stable real‑time computing platform using Spark Streaming on YARN, covering technical selection, architecture components, version compatibility challenges, exactly‑once semantics, graceful shutdown, Kafka tuning, and future enhancements.

Big DataExactly-OnceKafka
0 likes · 11 min read
How Lianjia Built a Low‑Latency Real‑Time Data Platform with Spark Streaming
Architecture Digest
Architecture Digest
Dec 27, 2017 · Backend Development

Handling Transactions, Failover, and Exactly‑Once Semantics in Distributed Systems

This article explores how distributed systems determine node liveness, manage failover and recovery, and implement at‑most‑once, at‑least‑once, and exactly‑once processing guarantees—including opaque transactions and two‑phase commit—using examples from Kafka, Zookeeper, and big‑data pipelines.

Big DataDistributed SystemsExactly-Once
0 likes · 15 min read
Handling Transactions, Failover, and Exactly‑Once Semantics in Distributed Systems
Meituan Technology Team
Meituan Technology Team
Nov 16, 2017 · Big Data

Performance Comparison of Apache Flink and Apache Storm for Real-Time Stream Processing

The study benchmarks Apache Flink against Apache Storm on a shared cluster, showing Flink delivering three‑to‑five times higher throughput and roughly half the latency across simple, sleep‑induced, and windowed workloads, with modest throughput loss for exactly‑once semantics, leading to a recommendation of Flink for high‑performance, stateful real‑time stream processing.

Apache FlinkApache StormExactly-Once
0 likes · 19 min read
Performance Comparison of Apache Flink and Apache Storm for Real-Time Stream Processing
Ctrip Technology
Ctrip Technology
Sep 20, 2017 · Big Data

Building a Real‑Time Computing Platform with Spark Streaming at Ctrip: Design, Implementation, and Lessons Learned

This article describes how Ctrip migrated its large‑scale real‑time platform from JStorm to Spark Streaming, detailing the architectural design, the Muise Spark Core encapsulation, operational metrics, encountered pitfalls, and future plans to adopt Flink and Beam for streaming workloads.

Big DataExactly-OnceSpark Streaming
0 likes · 22 min read
Building a Real‑Time Computing Platform with Spark Streaming at Ctrip: Design, Implementation, and Lessons Learned
Architecture Digest
Architecture Digest
May 14, 2017 · Big Data

Handling Transactions, Failover, and Exactly‑Once Semantics in Distributed Systems

This article explores practical techniques for handling node liveness, failover, recovery, and exactly‑once transaction semantics in distributed systems, illustrating implementations with Zookeeper, Kafka, Storm, and database sharding while addressing big‑data reach calculations and performance trade‑offs.

Big DataDistributed SystemsExactly-Once
0 likes · 15 min read
Handling Transactions, Failover, and Exactly‑Once Semantics in Distributed Systems