This article explains why Kafka can handle up to 20 million messages per second and 600 MB/s throughput by detailing producer batching and custom protocols, broker page‑cache, file layout and zero‑copy techniques, as well as consumer group strategies for efficient message consumption.
This article explains the four key techniques—page‑cache usage, sequential disk writes, zero‑copy transfers, and partitioned segment indexing—that enable Kafka to achieve exceptionally high write performance, detailing how each mechanism reduces latency and maximizes throughput.
Fury is a JIT‑based native multi‑language serialization framework that automatically handles shared and cyclic references, offers zero‑copy support, and delivers 20‑200× speed improvements over existing solutions, making it a high‑performance drop‑in replacement for Java, Python, Go, and C++ serialization needs.
This article explains Kafka's core concepts—including topics, partitions and replicas, log segment storage, leader‑follower mechanics, consumer groups, network threading model, zero‑copy I/O, and the essential role of Zookeeper for broker, topic, consumer, and offset management—providing a comprehensive overview for developers and architects.
This article explains the differences between buffers and caches, explores MySQL’s buffer pool architecture, details write‑through/write‑back strategies, and reviews key InnoDB parameters such as innodb_flush_log_at_trx_commit and innodb_flush_method for optimizing data durability and performance.
This article provides a comprehensive guide to Netty, covering its definition, advantages, key features, application scenarios, performance characteristics, architectural components, threading models, code examples, handling of packet framing, zero‑copy techniques, connection management, object pooling, and supported serialization protocols.
This article breaks down Kafka’s high‑availability, high‑performance, and high‑concurrency design, covering controller and leader election, replica and ISR mechanisms, ACK settings, the Reactor NIO model, zero‑copy I/O, compression, producer batching, memory‑pooling, and the multi‑layer network threading architecture.
This article explains Linux memory metrics VSZ, RSS, and PSS, illustrates them with a roommate analogy, details PageCache and its role in I/O, describes swap behavior and the swappiness setting, and introduces zero‑copy techniques to reduce data copying and context switches.
This article explains how Linux zero‑copy mechanisms such as DMA, sendfile, mmap and Direct I/O reduce data copies and context switches, detailing their operation, advantages, limitations, and real‑world usage in systems like Kafka and MySQL.
This article explains Java NIO’s non‑blocking I/O model, compares it with traditional BIO, details zero‑copy techniques, and walks through core components such as Channel, Buffer, and Selector with concrete code examples and diagrams, then shows how Redis leverages I/O multiplexing for high performance.
This article provides a comprehensive overview of Remote Direct Memory Access (RDMA), covering its definition, how it differs from traditional networking, core advantages such as zero‑copy and CPU offload, typical use cases, the three main RDMA protocols, deployment requirements, and essential programming concepts and terminology.
This article explains how DMA offloads memory‑to‑device data transfers, reduces the four data copies and context switches of traditional I/O, and introduces zero‑copy methods such as sendfile, mmap and Direct I/O, illustrating their mechanisms, advantages, limitations, and real‑world use cases like Kafka and MySQL.
Traditional Linux I/O using read() and write() involves multiple CPU and DMA copies and context switches, while modern techniques like zero‑copy, multiplexing, and page‑cache optimization reduce overhead; this article explains the classic I/O flow, read/write operations, network and disk I/O, and high‑performance strategies such as zero‑copy and Direct I/O.
This article presents a collection of interview questions covering backend fundamentals such as linked‑list sorting, encryption differences, TCP reliability, IO models, Hystrix, delayed tasks, HTTPS flow, transaction isolation, index pitfalls, virtual memory, Redis leaderboards, distributed locks, zero‑copy techniques, Java synchronized, and Snowflake ID generation.
The article presents a Guard abstraction that temporarily borrows and returns Protobuf field pointers via set_allocated/release, eliminating costly CopyFrom operations in recommendation pipelines, enabling zero‑copy field sharing across central‑control and recall stages, improving CPU usage and latency while handling safety and rollback.
This article explains Kafka's internal topics, preferred replicas, partition assignment processes, log directory layout, index files, offset and timestamp lookup, log retention and compaction policies, storage architecture, delayed operations, controller role, legacy consumer design flaws, rebalance workflow, and producer idempotence, providing a comprehensive overview of Kafka's backend architecture.
This article explains the costly four‑copy, four‑context‑switch data path in traditional Linux I/O, introduces DMA as a co‑processor that offloads memory transfers, describes zero‑copy techniques such as sendfile, mmap and Direct I/O, and shows how Kafka and MySQL leverage these methods to reduce CPU overhead and improve throughput.
This article explains why a message system like Kafka is crucial for decoupling services, handling asynchronous workflows such as e‑commerce flash sales, controlling traffic, and achieving high concurrency, high availability, and high performance through sequential disk writes, zero‑copy reads, replication, and careful resource planning.
This article explains how Kafka achieves remarkable speed and massive throughput by using sequential disk I/O, OS page cache, zero‑copy transfers, partitioned log segments with indexes, batch processing, and efficient compression, making it a cornerstone of modern big‑data pipelines.
Zero-copy techniques eliminate unnecessary data copying between user and kernel space, dramatically improving I/O performance; this article explains core I/O concepts, explores Java implementations like MappedByteBuffer, DirectByteBuffer, channel-to-channel transfers, and demonstrates Netty’s composite buffers, plus examples from Kafka and RocketMQ.
Zero‑copy I/O minimizes data copying by using DMA and kernel mechanisms such as mmap + write or the sendfile() system call, with sendfile achieving true zero‑copy through fewer system calls, context switches, and memory copies compared to traditional read/write paths.
Kafka’s I/O efficiency stems from sequential data writes, zero‑copy reads using sendfile, batch compression of messages, and a dual‑threaded batch producer that together minimize random disk access and data copying, dramatically speeding up both reading and writing operations.
This article provides a comprehensive technical guide to Kafka performance, covering the core bottlenecks of network, disk and complexity, detailing optimization techniques such as concurrency, compression, batching, caching and algorithms, and explaining how Kafka’s sequential write, zero‑copy, page cache, reactor‑based network model, batch handling, partition concurrency, and file structure contribute to high throughput.
This article explains how zero‑copy techniques such as mmap, sendfile, and splice reduce CPU involvement in data transfer by avoiding memory copies between kernel and user spaces, and shows how Java NIO and Netty implement these mechanisms for high‑performance backend I/O.
This article explains how Netty, an asynchronous event‑driven Java NIO framework, simplifies building high‑performance, maintainable network servers and clients by handling I/O, threading, and protocol concerns, and it covers its thread models, zero‑copy techniques, common interview questions, and real‑world use cases such as Dubbo and RocketMQ.
Zero-copy techniques such as mmap + write, sendfile, and sendfile + DMA scatter/gather reduce CPU copies and context switches by leveraging kernel‑space data transfers and DMA, dramatically improving I/O performance for high‑throughput systems like RocketMQ and Kafka, as explained with detailed process flows and comparisons.
This article explains the concepts of user space and kernel space, shows how traditional file‑to‑socket transfers involve multiple data copies with read/write system calls, and demonstrates how the Linux sendfile system call implements zero‑copy to reduce copies and system calls for high‑performance servers.
This article explains the concept of zero‑copy, its performance benefits, and how Java NIO, memory‑mapped buffers, direct buffers, channel‑to‑channel transfers, and Netty’s composite buffers implement zero‑copy to reduce data copying between kernel and user space.
This article provides a comprehensive overview of Kafka performance optimization, covering network, disk, and complexity challenges, sequential write techniques, zero‑copy data transfer, page‑cache usage, reactor‑based networking, batch processing, compression, partition concurrency, and file‑structure design to help developers build high‑throughput, low‑latency streaming systems.
The article explains Linux zero‑copy I/O techniques—mmap, sendfile, splice, and tee—detailing their design principles, execution flows, context‑switch and data‑copy reductions, advantages, limitations, and appropriate use cases, helping developers choose the optimal method for high‑performance file and network transfers.
This article explains the purpose of message systems, compares various solutions such as RabbitMQ, Redis, ZeroMQ, ActiveMQ, RocketMQ and Kafka, then details Kafka's design goals, core concepts, architecture, replication, retention policies, zero‑copy transfer, batching, and performance optimizations for high‑throughput distributed messaging.
This article explains the concept of zero-copy I/O, covering buffer types, virtual memory, mmap+write and sendfile techniques in Java, and demonstrates how Java NIO, Netty, and other frameworks like RocketMQ and Kafka leverage zero-copy to reduce data copying and improve performance.
This article explains how Kafka attains million‑level transactions per second by using sequential disk reads/writes, memory‑mapped files, DMA‑based zero‑copy transfers, and batch data transmission, detailing each technique and its impact on throughput and latency.
Kafka can sustain millions of transactions per second by writing data sequentially to disk, leveraging memory‑mapped files, employing zero‑copy DMA transfers, and batching messages, each technique reducing I/O overhead and CPU involvement, which together enable its high‑throughput performance in big‑data pipelines.
Zero-copy techniques eliminate unnecessary data copying between user and kernel space, dramatically improving I/O performance; this article explains core concepts, Java NIO’s MappedByteBuffer and DirectByteBuffer, channel-to-channel transfers, Netty’s composite buffers, and how frameworks like Kafka and RocketMQ leverage zero-copy.
This article explains the zero‑copy data transfer technique, compares it with traditional read/write approaches, shows Java NIO code examples, and discusses its use in high‑performance systems such as Kafka and Spark, highlighting the reductions in context switches and memory copies.
This article provides a comprehensive overview of Java IO, explaining the traditional blocking BIO model, the modern non‑blocking NIO architecture, the role of streams, buffers, channels, zero‑copy techniques, and selectors, and includes practical code examples for file copying and simple client‑server communication.
The article explains how Kafka achieves high throughput by using partition‑level parallelism, sequential disk writes with segment files, extensive use of the OS page cache, zero‑copy data paths, request batching and optional compression, while also discussing the underlying disk I/O principles.
This article explains Linux's zero‑copy techniques—mmap, sendfile, and splice—detailing how they reduce data copies between user and kernel space, improve I/O efficiency, handle edge cases like SIGBUS, and leverage DMA to minimize CPU overhead in server file transfers.
This article explains the zero‑copy technique in Java, covering basic I/O concepts, buffer management, mmap + write and Sendfile methods, and how frameworks like NIO, Netty, Kafka, and RocketMQ use these mechanisms to eliminate data copies and improve performance.
This article explains why traditional read/write file serving incurs multiple data copies and context switches, then introduces mmap, sendfile, and splice as zero‑copy methods that reduce CPU load, discuss their usage, limitations, and practical pitfalls for high‑performance Linux servers.
This article explains the principles of zero‑copy I/O, covering buffer concepts, virtual memory, mmap + write and sendfile methods, and demonstrates Java implementations using MappedByteBuffer, DirectByteBuffer, channel‑to‑channel transfers, as well as Netty’s CompositeChannelBuffer for efficient data handling.
This article explains the principles of Linux I/O, the need for zero‑copy, the role of page and buffer caches, various data‑transfer mechanisms such as DMA, Direct I/O and mmap, and compares their performance and usage scenarios.
This article explains how Netty achieves high‑performance, scalable networking by addressing the C10K and C10M problems through efficient I/O models, multithreading, zero‑copy ByteBuf handling, memory allocation strategies, and resource management techniques, providing practical code examples and architectural diagrams.
This article explains how zero‑copy techniques such as DMA, mmap, and sendfile work, compares them with traditional file I/O, shows their Java NIO implementations, and presents performance test results demonstrating the advantages of mmap for small files and sendfile for large files.
This article explains the fundamentals of zero‑copy I/O, covering buffer and virtual memory concepts, Java NIO mechanisms such as MappedByteBuffer and DirectByteBuffer, channel‑to‑channel transfers, and how Netty implements zero‑copy with composite buffers, providing code examples and diagrams for each technique.
This article explains the zero‑copy principle, describes core I/O concepts such as buffers and virtual memory, and details Java implementations including mmap + write, Sendfile, MappedByteBuffer, DirectByteBuffer, channel‑to‑channel transfer, and Netty’s composite buffers for high‑performance data transfer.
Kafka achieves its remarkable speed by combining sequential I/O, batch processing, compression, zero‑copy, careful client‑side work, and a design that avoids costly fsync and garbage collection, while maintaining durability, ordering, and at‑least‑once delivery, making it a high‑throughput, low‑latency event streaming platform.
This article explains the concept of zero‑copy, how Netty implements it through sendfile, FileChannel.transferTo and CompositeChannelBuffer, and provides detailed code analysis of CompositeChannelBuffer's internal structures and lookup algorithms to improve network I/O performance.
This article explains how to bypass the costly GPU‑CPU‑GPU copy when using Flutter external textures by leveraging CVPixelBuffer's shared memory mechanism, detailing the underlying engine changes, registration and rendering steps, code implementation, and a demo that achieves smooth 60 FPS rendering.
Zero‑copy techniques in Linux eliminate unnecessary data copying between user and kernel space, dramatically reducing CPU overhead and boosting throughput; this article explains why zero‑copy is needed, outlines several methods—including user‑space I/O, mmap, sendfile, DMA‑assisted sendfile, splice, copy‑on‑write, buffer sharing, and netmap—plus their trade‑offs.
Kafka achieves its ultra‑high throughput and low latency by writing data to the OS page cache, performing sequential disk writes, and employing zero‑copy techniques that eliminate unnecessary data copies during consumption, enabling tens of thousands to millions of messages per second.
This article explains the traditional copy‑based data transmission process, introduces the zero‑copy technique—including basic sendfile(), scatter/gather DMA and mmap support—shows how it reduces context switches and copies, and demonstrates its practical use in Kafka and Spark for high‑throughput workloads.
This article explains how Kafka attains high throughput by using sequential disk writes, memory‑mapped files, sendfile zero‑copy, and batch compression, detailing both write and read path optimizations and their impact on performance.
This article explains the concept of zero‑copy, its I/O fundamentals, and how Java, Netty, RocketMQ and Kafka use mmap, sendfile and other techniques to eliminate data copies and boost backend throughput and latency.
Meituan’s DSP system boosted high‑QPS ad serving performance by layering an LRU cache in front of Redis, then adding time‑based eviction, sharding the cache into HashLruCache instances to cut lock contention, and employing a zero‑copy, reference‑counted design, ultimately cutting average latency to about 20 % of the original and similarly reducing 99.9th‑percentile delays.
This article explains Linux's separation of user and kernel address spaces, details the five classic network I/O models—blocking, non‑blocking, I/O multiplexing, signal‑driven, and asynchronous—and introduces zero‑copy techniques such as mmap, sendfile and splice to reduce data copying overhead.
This article explains the lifecycle of data in I/O operations, compares blocking, non‑blocking, and asynchronous models, introduces zero‑copy techniques, and details how select, poll, and epoll enable efficient multiplexed I/O handling in server applications.
This article explains the growing need for more efficient data‑center networking, introduces Remote Direct Memory Access (RDMA) technology, describes its working principles, operation types, and how it can be layered over TCP/Ethernet to reduce latency and CPU overhead in high‑performance environments.
This article explains how Kafka’s design—using sequential disk reads/writes, zero‑copy system calls, file segmentation, batch sending, and message compression—delivers massive throughput while minimizing performance loss and network load.
Zero‑copy, implemented via the Linux sendfile system call, eliminates data copying between kernel and user space, reducing context switches and memory usage, thereby significantly improving the performance of web servers such as Apache and Nginx.
