This article explains Kafka's core architecture—including topics, producers, brokers, and consumers—its underlying mechanisms, the role of Zookeeper, key characteristics such as high throughput and fault tolerance, and common use cases like log collection, activity tracking, and stream processing.
Using Go’s pprof, the team traced a gradual CPU spike in a high‑throughput kafka‑go consumer to a saturated commit queue and repeatedly nested context values, which forced costly lookups; eliminating the unnecessary trace‑id context injection (or recreating a fresh context each loop) resolved the issue and reduced CPU usage to under 2 %.
This article explains how to enable cross‑node WebSocket communication in a distributed Spring application by publishing messages to a Redis or Kafka topic, tracking user connections, and routing messages to the appropriate server instance, complete with full code examples and configuration details.
This article details a real‑world investigation of severe latency in CKafka cross‑region data synchronization, tracing the issue from high message backlog through network bandwidth tests, kernel parameter adjustments, and finally uncovering a TCP window‑scaling failure caused by SYN‑cookie protection and missing timestamp options.
This article presents Ctrip's practical experience of constructing an offline‑online data warehouse, detailing business pain points, objectives, system architecture, component design, data quality measures, and future directions to achieve scalable, real‑time data processing and management.
This article deeply analyzes Kafka broker’s network architecture and request‑handling pipeline, walking through simple sequential models, multithreaded async designs, the Reactor pattern with Java NIO, key thread roles, core processing flow, and practical tuning parameters for high‑throughput, low‑latency deployments.
This article outlines practical methods for ensuring timeliness, data quality, stability, cost efficiency, agility, and management in real‑time data warehouse pipelines using technologies like Flink and Kafka, while addressing consistency, completeness, and high‑availability concerns.
The article chronicles Netflix’s evolution from a failing batch pipeline to a cloud‑native, self‑service streaming platform, detailing four development phases, the technical challenges faced, the stream‑processing patterns introduced, key learnings, and future opportunities for real‑time data and machine‑learning workloads.
This article traces the development of message queues from early decoupling solutions like ActiveMQ and RabbitMQ, through high‑throughput designs such as Kafka and RocketMQ, to modern platform‑centric systems like Pulsar, while detailing core concepts, architecture diagrams, storage mechanisms and trade‑offs.
This article breaks down how Kafka achieves up to 20 million messages per second and 600 MB/s throughput by examining producer batching and compression, broker PageCache and zero‑copy techniques, and consumer parallelism with group coordination.
The article traces two decades of message‑queue evolution—from early decoupling tools like ActiveMQ, through Kafka’s high‑throughput log model, to Pulsar’s layered, cloud‑native architecture—explaining core concepts, storage designs, and trade‑offs that guide choosing the right MQ for modern workloads.
The article details NetEase Cloud Music’s Corona monitoring pipeline, explaining how SDKs report logs via an HTTP service, how a transmission layer normalizes and stores them, how a Flume‑like collector forwards logs to HBase and Kafka, and how Flink tasks shard and filter streams for various monitoring services while handling traffic spikes and offering an independent Node.js channel for other business units.
This article explains how large hotel‑booking platforms such as Airbnb, Booking.com and OYO use a micro‑service architecture—including hotel management, customer search/booking, and view‑booking services—combined with load balancers, Kafka, Elasticsearch, Redis, Cassandra and Hadoop to achieve a seamless, high‑throughput booking flow.
This article explains how Cloudflare’s engineering teams built and evolved a highly scalable Kafka‑based messaging backbone, adopted protobuf for structured communication, created a generic message‑bus, implemented connectors, added observability with Prometheus and OpenTelemetry, and refined health‑checks and batch processing to support trillions of inter‑service messages.
This article presents a comprehensive design and implementation of a serverless delayed message queue on AWS, evaluating various solutions such as RabbitMQ, ActiveMQ, RocketMQ, Redis, and ultimately proposing a hybrid Kafka‑SQS‑DynamoDB architecture with performance optimizations and operational insights.
The article presents a detailed three‑month interview journey that led to a position at a top new‑energy automotive firm, outlining the questions and topics covered in five interview rounds—including Hive, Spark, Flink, Kafka, data modeling, and data governance—to help candidates prepare for big‑data roles.
This article provides a comprehensive overview of Kafka, covering its core architecture, message queue models, communication process, partition selection, consumer groups, rebalancing strategies, partition assignment algorithms, reliability guarantees, replica synchronization, and reasons for removing Zookeeper in newer versions.
Kafka is an open‑source, distributed streaming platform that uses a publish/subscribe message queue architecture to provide high‑throughput, fault‑tolerant real‑time data processing, featuring topics, partitions, replicas, consumer groups, and multiple APIs for producers, consumers, streams, connectors, and administration.
This article provides a comprehensive overview of Kafka's architecture and fundamental concepts, covering its overall structure, key components such as brokers, producers, consumers, topics, partitions, replicas, leader‑follower synchronization, offset handling, message storage at both logical and physical layers, as well as producer and consumer workflows, partition assignment strategies, rebalancing, log management, zero‑copy I/O, and reliability mechanisms.
This article provides a detailed overview of Apache Kafka, covering its distributed streaming architecture, storage mechanisms, replication, consumer groups, compression techniques, exactly‑once semantics, configuration tips, and performance optimizations for building reliable high‑throughput data pipelines.
Didi’s unified real‑time data stack outlines best‑practice component choices for four key scenarios—metric monitoring, BI analysis, online services, and feature/tag systems—detailing pipelines from source to sink, resource‑usage guidelines, and a one‑stop development platform to build stable, high‑performance streaming solutions.
This article provides a comprehensive overview of Kafka, covering its core value for decoupling and asynchronous processing, fundamental concepts such as producers, consumers, topics, partitions and replication, high‑performance mechanisms like zero‑copy and OS cache, detailed resource evaluation for CPU, memory, disk and network, operational tools, consumer‑group rebalance strategies, LEO/HW offsets, controller management, and delayed‑task scheduling.
This article comprehensively explores Kafka’s core concepts, architecture, and the techniques it employs—such as ack strategies, replica synchronization, high‑watermark, leader‑epoch, zero‑copy, batch sending, compression, and reactor‑based networking—to achieve both strong reliability and high throughput in distributed messaging systems.
A multi‑expert panel shares experiences, challenges, and best practices for building, operating, and evolving large‑scale event‑driven systems using technologies like Kafka, covering architecture decisions, domain modeling, observability, handling unordered events, and advice for day‑two operations.
This article examines synchronous versus asynchronous microservice communication, outlines the benefits of async messaging, and compares three popular message brokers—RabbitMQ, Kafka, and Redis—by evaluating their scale, persistence, consumer models, and ideal use cases to help developers choose the right solution.
This article explains how to design and implement a basic task system—covering task configuration, display, progress tracking, and reward collection—while using Kafka for decoupling services and a publish/subscribe event layer to keep task logic independent of event sources.
This article demonstrates how to use the Go Sarama library to configure Kafka producers and consumers, provides sample code for sending and receiving messages, and includes a lightweight performance test that repeatedly publishes messages to evaluate throughput and latency.
The article explains how to archive Kafka real‑time data to Hive using either Flink SQL, which quickly creates partitioned ORC tables but requires timezone handling, or Flink DataStream for more complex pipelines, and offers best‑practice guidance on data quality, system complexity, security, and performance.
This article presents a comprehensive overview of NetEase Cloud Music's real‑time data warehouse architecture, its stream‑batch consistency model, the low‑code FastX platform implementation, and future plans for expanding lake‑house capabilities and high‑availability solutions.
This article shares a recent graduate's successful job offer story, emphasizes preparing a detailed personal project summary, and provides extensive big‑data interview questions covering Hadoop, Spark, Flink, Kafka, Hive, ClickHouse, and related technologies to help candidates excel in interviews.
This article compiles extensive big‑data interview questions from companies like Bilibili, ByteDance, Ant Group, and Tencent, offers practical advice on project depth, open‑source contributions, and provides strategic insights for recent graduates navigating a tightening job market.
This article provides an in-depth overview of Kafka, covering its core value as a message queue, fundamental concepts, cluster architecture, producer and consumer configurations, scaling strategies, monitoring tools, and practical operational commands for building and maintaining high‑throughput, highly available streaming systems.
This article explains Netflow’s principles, its three‑component architecture, implementation details using open‑source tools like nfdump and Kafka, and showcases practical applications such as load balancing, anomaly detection, and traffic engineering, providing a comprehensive guide for building robust network flow monitoring solutions.
This article explains how to configure Spring Boot 2.7.10 for AMQP‑based RabbitMQ and Apache Kafka messaging, covering property settings, connection factories, template customization, sending and receiving messages, listener containers, retry mechanisms, and stream processing with code examples.
This article presents NetEase Cloud Music's real-time data warehouse architecture, covering its streaming and batch scenarios, layered design (ODS, CDM, ADS), technology stack choices, consistency mechanisms, the FastX low-code platform, and future development plans, offering a comprehensive technical overview for data engineers and architects.
This article introduces Kafka, a high‑performance distributed messaging system, and provides step‑by‑step Java/Groovy examples for configuring a producer and consumer, demonstrating how to benchmark their throughput using the FunQpsConcurrent framework, along with necessary Gradle dependencies and server setup instructions.
This article provides a comprehensive overview of Kafka, covering its core value as a message queue, fundamental concepts, cluster architecture, log storage mechanisms, zero‑copy data transfer, high‑throughput and high‑availability design, consumer group behavior, rebalance strategies, and practical operational commands for managing topics, partitions, and offsets.
This article presents Xiaohongshu's design and implementation of a unified streaming‑batch storage system that integrates Lambda architecture, Kafka, Flink, Iceberg, and modern OLAP engines to solve real‑time data warehouse pain points and enable consistent, exactly‑once analytics across streaming and batch workloads.
Vivo’s push platform achieves high‑availability disaster recovery by deploying multi‑region broker clusters, implementing dual‑active logic nodes across two data centers, adding a Kafka‑backed buffering layer for traffic spikes, and using a hybrid Redis‑plus‑disk KV storage scheme to ensure durable, real‑time message delivery.
This article explains the purpose, semantics, and design of Apache Kafka's transaction API, describes how exactly‑once processing is achieved in stream‑processing applications, outlines the Java client usage, and discusses the internal components, performance considerations, and best‑practice tips for developers.
This article explains Kafka's cross‑data‑center high‑availability options, compares stretched and connected cluster designs, outlines typical failure scenarios, and reviews both community and commercial replication solutions, helping architects choose the most suitable deployment for their specific requirements.
This article provides a comprehensive technical overview of Kafka, covering its core message‑queue value, architecture components such as producers, consumers, topics, partitions and replication, high‑performance mechanisms like zero‑copy and OS cache, resource planning for disks, memory, CPU and network, operational tools and commands, consumer‑group management, rebalance strategies, and internal scheduling mechanisms such as the time‑wheel.
This article details ByteDance's end‑to‑end design and engineering of a low‑overhead, continuous profiling system that collects, processes, and analyzes resource usage across a million‑scale data center, enabling precise performance optimization and significant CPU cost savings.
This article describes the design, implementation, and evolution of Heimdall, an exception‑statistics platform built on Kafka, Flink, and HBase that provides minute‑level anomaly aggregation, stack trace querying, and integration with release and alerting workflows to improve service reliability across thousands of micro‑services.
This article provides a comprehensive overview of major message‑queue middleware—including RocketMQ, Kafka, Pulsar, and RabbitMQ—covering fundamental concepts such as tags, groups, offsets, architectural components, storage mechanisms, transaction workflows, rebalance strategies, and recent developments, while comparing their features and performance characteristics.
The article explains the AKF Scale Cube model—horizontal replication (X axis), functional decomposition (Y axis), and data/service partitioning (Z axis)—and demonstrates how these three scaling dimensions can be applied to backend systems such as Kafka and Redis to achieve high availability, performance, and fault isolation.
This article provides a detailed side‑by‑side comparison of five popular message‑queue systems—Kafka, RabbitMQ, ZeroMQ, RocketMQ, and ActiveMQ—covering documentation, programming languages, supported protocols, storage, transactions, load balancing, clustering, management UI, availability, duplication handling, throughput, subscription models, ordering, acknowledgments, replay, retry, concurrency, and includes promotional information from the author.
This case study describes how Hangzhou Mingshitang transformed its homework photo evaluation system from a Kubernetes‑based solution to an Alibaba Cloud Function Compute serverless architecture, achieving higher elasticity, lower latency, and reduced costs while handling peak traffic of over one million daily images.
This article presents a comprehensive design and implementation of a real‑time cache update system that leverages Kafka‑driven database change streams, a centralized cache scheduling center, executor registration, broadcast and fail‑over scheduling, and a lightweight SDK to achieve millisecond‑level cache consistency for C‑end services.
This article explains Kafka’s core concepts, architecture, and the mechanisms—such as ACK policies, replication, HW/LEO management, zero‑copy I/O, batching, compression, and load‑balancing—that together ensure high reliability and high throughput for asynchronous decoupling and peak‑shaving scenarios.
This article explains why MQ middleware like Kafka is introduced for system decoupling and traffic control, outlines the three key challenges of message loss detection, loss points, and prevention, and provides detailed configurations, monitoring tips, and code examples to ensure reliable, loss‑free message delivery.
This article explains why message duplication occurs in Kafka, describes the three delivery semantics, and provides practical solutions—including idempotent producers, transactions, and consumer-side idempotence—along with configuration tips and code examples to achieve exactly‑once delivery.
This article explains the common interview question of guaranteeing 100% message reliability in MQ middleware such as Kafka or RabbitMQ, outlines the three lifecycle stages of a message, discusses detection mechanisms, id generation, idempotent consumption, and handling message backlog, providing concrete design patterns and practical examples.
ByteHouse evolved its real‑time import pipeline from a community ClickHouse architecture to a custom HaKafka engine and a cloud‑native design, addressing node failures, read‑write conflicts, scaling costs, and latency by introducing two‑level concurrency, memory tables, exactly‑once semantics, and robust fault‑tolerance.
This article details the motivation, architectural evolution, and technical implementations of ByteHouse's real‑time ingestion pipeline, covering internal business requirements, distributed‑system challenges, the custom HaKafka engine, memory‑table optimizations, and the transition to a cloud‑native design that delivers high availability, low‑latency, and exactly‑once semantics.
This article provides a comprehensive overview of Kafka’s core concepts, high‑performance design, cluster planning, resource evaluation, deployment steps, producer and consumer configurations, fault‑tolerance mechanisms, and operational tools, offering practical guidance for building and managing a high‑throughput Kafka production environment.
Echo is a full‑stack open‑source Java community system built with Spring Boot, MyBatis, MySQL, Redis, Kafka and Elasticsearch, offering modules like posts, comments and notifications, and the article provides its core tech stack, development environment, local setup steps, deployment architecture, demo screenshots and source code access.
This article explains how to design and implement a real‑time, multi‑tenant site search system by migrating data from MySQL to Elasticsearch using Kafka and MySQL binlog, covering architecture decisions, data schema, Docker deployment, and core SpringBoot/Kotlin code for event collection and indexing.
This guide explains why message queues are introduced for decoupling and traffic control, identifies three key areas where message loss can occur—in producers, brokers, and consumers—and provides concrete Kafka configurations, monitoring practices, and operational steps to ensure reliable, loss‑free message delivery.
This article explains why exactly‑once semantics are needed for stream‑processing applications, describes Kafka's transactional model and semantics, details the Java transaction API and its usage, and discusses the internal components, performance trade‑offs, and practical guidelines for building reliable Kafka‑based pipelines.
This article demonstrates how to achieve exactly‑once processing in Flink by providing Kafka I/O utility classes, a character‑count streaming example, and a transactional consumer implementation, while also discussing configuration nuances and common pitfalls.
A production logging system became unavailable due to Kafka backlog alerts, prompting an investigation that uncovered read‑only ClickHouse tables caused by mismatched Zookeeper metadata after a TTL policy change, leading to a step‑by‑step recovery involving Zookeeper restarts, metadata fixes, and table reconstruction.
This tutorial explains how to achieve exactly‑once semantics when streaming data from Kafka to Redis using Apache Flink's TwoPhaseCommitSinkFunction, covering Redis transaction basics, utility classes, sink implementation, testing steps, and solutions to common connection and transaction bugs.
The article explains how market trends, data‑driven enterprise needs, and internal platform experience led Didi to build Know Streaming—a zero‑intrusion, plugin‑based real‑time streaming solution built on Kafka—to address scalability, operability, and community adoption challenges.
This article describes the challenges of a highly complex supply‑chain system, the evolution from early MySQL‑based reporting to a modern real‑time data platform using Flink, Kafka, ClickHouse, Hologres and other cloud services, and the tools and lessons learned to achieve low‑latency, high‑throughput analytics.
Facing chaotic usage, maintenance challenges, partition tolerance issues, and performance bottlenecks with RabbitMQ, our middleware team decided to fully migrate to Kafka, outlining reasons, comparative models, migration strategies, and verification steps to ensure a smooth, high‑availability, high‑performance transition.
This article explains Kafka's core concepts—including messages as byte arrays, optional keys for partition control, topic and partition organization, producer and consumer roles, offsets, consumer groups, and broker clusters—providing a concise technical overview for developers learning Kafka.
This article explains how to configure Flink SQL to emit early‑fire results for tumbling windows, allowing real‑time aggregation of metrics like PV and UV, and provides complete example code, execution output, and a discussion of current limitations.
The article demonstrates how to use Spring's HandlerMethodArgumentResolver to automatically convert Kafka messages into typed method parameters, eliminating repetitive JSON parsing code and showing a complete implementation, usage example, and notes on version support and performance considerations.
This article explains how Kafka's Schema Registry manages schema evolution and compatibility types—backward, forward, transitive, and full—using Avro schemas, demonstrates the impact of field additions or deletions on producers and consumers, and shows how to change a topic's compatibility setting via REST API.
This article explains why message duplication occurs in Kafka pipelines, describes the three delivery semantics, and provides three practical solutions—Kafka idempotent producer, Kafka transactions, and consumer‑side idempotence—along with configuration tips and code examples.
This article explains the concept of retract streams in Apache Flink, detailing how non‑retract Kafka sources and Group‑By aggregations generate delete/insert messages, provides code examples for aggregation and sink operators, and compares retract mechanisms across aggregation and CDC sink scenarios.
This article presents Youzu Network’s adoption of StarRocks for multi-dimensional analytics, detailing the historical OLAP challenges, StarRocks’ features and advantages, its application scenarios, data modeling choices, ingestion methods, performance benchmarks, and future roadmap for unified analytics.
This article summarizes Liu Chong's DTCC2022 talk on building a real‑time advertising data warehouse, covering business context, layered model design, development technologies such as Flink and Kafka, full‑link quality assurance, practical implementation details, and future architectural directions.
An incident report details how a scheduled machine reboot on Alibaba Cloud triggered a Flink TaskManager failover, leading to excessive data replay, increased ES pressure, and significant business latency, and explains the root cause involving disabled checkpoints and timestamp‑based offset consumption.
This article explores strategies for batch scaling Kafka replicas with minimal data migration, presenting two design ideas, detailed calculations of broker lists, partition counts, start indexes, and replica shifts, and provides step‑by‑step algorithms and code snippets to compute optimal replica assignments for both expansion and contraction scenarios.
vivo’s Internet Storage team details how it operates RocketMQ for low‑latency online services and Kafka for massive big‑data pipelines, outlines resource isolation, traffic balancing, intelligent throttling, and governance practices, and describes its migration from RabbitMQ and planned shift from Kafka to cloud‑native Pulsar.
This article explains Kafka's multi‑replica design, ISR mechanism, leader election rules, and producer acknowledgment settings, then shows how the built‑in __consumer_offset topic with a single replica can cause a whole cluster to become unavailable when one broker crashes, and offers practical fixes.
This article details Meituan's use of Kafka as a unified data cache and distribution layer, outlines the challenges of massive scale and latency, and presents comprehensive optimizations across application, system, and cluster management layers, including disk balancing, migration acceleration, fetcher isolation, and full‑link monitoring.
This comprehensive guide explores Kafka as a distributed messaging middleware, detailing its core concepts, architecture, producer and consumer mechanisms, configuration options, Zookeeper integration, controller responsibilities, network model, performance optimizations such as zero‑copy, page‑cache usage, batching, compression, and partition concurrency.
This article introduces the fundamentals of message middleware, outlines its core components, and compares RocketMQ and Kafka architectures to show how they achieve high‑availability through replication, leader election, and coordinated producer‑consumer interactions.
This article explains why real‑time data warehouses are becoming essential, outlines their goals, compares them with traditional offline warehouses, and presents detailed design patterns, naming conventions, and case studies from Didi, Kuaishou, Tencent, Youzan and other enterprises, highlighting challenges and solutions for streaming, storage, and query layers.
This article explains the concepts of virtual memory, kernel and user modes, DMA, and why data must move between kernel and user spaces, then compares standard I/O with zero‑copy techniques, MMAP, and off‑heap buffers, showing how they reduce copy operations and improve Java server performance.
This article presents the design principles, core techniques, and practical solutions of Vivo's self‑developed Bees log collection agent, covering file discovery, unique identification, real‑time and offline ingestion, checkpointing, resource control, platform management, and a comparison with open‑source alternatives.
This article presents the design principles, core features, and implementation details of Vivo's self‑developed Bees log collection agent, covering file discovery, unique identification, real‑time and offline ingestion, resource control, platform management, and comparisons with open‑source solutions.
This article details Meituan's large‑scale Kafka deployment—over 15,000 machines and petabyte‑level daily traffic—its operational challenges such as slow nodes, load imbalance, and resource contention, and the comprehensive read/write latency, system‑level, and cluster‑management optimizations implemented to improve performance and reliability.
This article provides an in-depth comparison of RabbitMQ, RocketMQ, and Kafka, detailing their core components, exchange types, message durability, acknowledgment mechanisms, TTL, dead‑letter queues, load balancing, ordering, transaction handling, high‑availability configurations, and practical solutions for common messaging challenges.
This guide outlines Kafka stability best practices across three phases—pre‑prevention with tuning, producer/consumer guidelines, and cluster configuration; runtime monitoring using white‑box and black‑box metrics and alerts; and fault resolution strategies for backlogs, consumption blocks, and message loss, plus cost control and idempotence techniques.
Vivo’s Bees‑agent is a custom, lightweight log‑collection service that discovers rotating files via inotify, uniquely identifies them with inode and hash signatures, supports real‑time and offline ingestion to Kafka and HDFS, offers checkpoint‑resume, resource isolation, rich metrics, and a centralized management platform, outperforming open‑source collectors in latency, memory usage, and scalability.
This article details the design, architecture, and implementation of Meituan's Logan real‑time logging platform, covering its workflow, multi‑terminal collection SDK, ingestion, Flink‑based processing, consumption layers, stability measures, and future roadmap, illustrating how it improves fault diagnosis and system reliability.
This article describes Liulishuo's migration to a Flink‑based real‑time data warehouse, covering background, benefits, technology selection (storage, Flink platform, dimension table connectors), overall architecture, concrete Hudi and Elasticsearch ingestion examples, processing SQL, and future outlook for unified batch‑streaming storage.
The article walks through the complete lifecycle of a Kafka consumer SyncGroupRequest, detailing request headers and bodies, coordinator selection, state handling on the GroupCoordinator, metadata persistence, and the client‑side response processing that transitions members to a stable state.
This article chronicles the author's four‑stage evolution with message queues—from early experiments with ActiveMQ, through Redis and RabbitMQ, to MetaQ and finally RocketMQ—highlighting practical challenges, architectural decisions, performance tuning, and insights for building robust, high‑throughput backend systems.
This article provides a comprehensive overview of Kafka consumer group rebalance, covering version compatibility, rebalance triggers, assignment strategies, generation handling, protocol details, the full rebalance workflow, listener usage, and complete Java code examples for offset management with database integration.
This article details Meituan's large‑scale Kafka deployment, describing the current state, performance challenges such as slow nodes and disk imbalance, and the comprehensive optimizations applied—including read/write latency reductions, migration pipelines, fetcher isolation, SSD caching, RAID acceleration, cgroup isolation, full‑link monitoring, service lifecycle management, and TOR disaster recovery—to improve reliability and prepare for future growth.
This article explains how to use Know Streaming’s zero‑intrusion plugin to query Kafka messages, detailing the UI flow, multi‑dimensional filter options such as offset, partition, and key/value selection, the data truncation feature, and the underlying implementation that builds a KafkaConsumer and leverages KafkaAdminClient.
This article describes how Meituan built Logan, a high‑performance, end‑to‑end real‑time logging platform for mobile, web, mini‑programs and IoT, covering its background, architecture, data collection, processing, consumption, monitoring, deployment strategies, achieved results and future roadmap.
This article explains how Kafka achieves extremely high throughput—up to 20 million messages and 600 MB per second per node—by optimizing the producer, broker, and consumer components through batch sending, custom protocols, page‑cache usage, zero‑copy transfers, and efficient compression algorithms.
This article thoroughly examines Kafka's storage system, explaining why it uses sequential log writes combined with sparse indexing, how different log formats evolved, and the mechanisms for log retention and compaction that enable high‑throughput, fault‑tolerant streaming at massive scale.
The article details Meituan’s end‑to‑end design and implementation of Logan, a high‑performance real‑time logging service for mobile apps, web, mini‑programs and IoT, covering background challenges, architecture layers, technology choices such as Flink and Elasticsearch, stability measures, deployment practices, achieved results and future plans.
