The article explains how Linux uses page cache, inode cache, and dentry cache to accelerate file operations, detailing their structures, lookup mechanisms, LRU eviction, and real‑world performance impact, and shows step‑by‑step path resolution with code examples.
The article explains how Linux’s page cache bridges memory and disk, detailing its read/write mechanisms, dirty page handling, pre‑read optimization, kernel parameters, and practical tuning tips for static file serving, databases, and logging, showing why mastering it is essential for performance.
This article breaks down Kafka's high‑throughput design by detailing how partitioning enables parallelism, sequential disk writes and zero‑copy reduce I/O overhead, and OS page cache maximizes memory efficiency, together allowing millions of messages per second.
This article explains how Kafka leverages sequential disk I/O, zero‑copy data transfer, OS page cache, and partition‑based parallelism to deliver massive throughput, detailing the underlying mechanisms, performance numbers, and a practical formula for estimating total system capacity.
This article explains why memory leaks act like invisible thieves that gradually fill the RSS space, outlines their four‑step attack process, shows how to spot the tell‑tale signs using process‑level and system‑level metrics, and provides practical emergency and preventive measures to protect your applications.
The article examines Linux's page‑cache writeback mechanism, explains why the single‑threaded writeback path becomes a bottleneck under heavy writes, and details Vivo's multithreaded writeback patches—including inode‑to‑context mapping and sysfs‑controlled thread counts—that achieve up to 2.4 GB/s on XFS and a 22 % speedup on F2FS, while also discussing fragmentation trade‑offs and optimal thread‑to‑allocation‑group ratios.
The article examines how Huawei’s ModelFS makes prefetch and memory reclamation programmable and how vivo adapts F2FS to support uncached buffer I/O, enabling AI model weights to be loaded without lingering in the page cache and eliminating kswapd overhead.
This article explains the core Kafka techniques—sequential disk writes, Linux page cache usage, producer/broker batching with compression, and zero‑copy data transfer via sendfile—that together enable massive concurrency and near‑memory write performance in large‑scale streaming architectures.
The article explains how Kafka attains high‑throughput performance by using sequential disk writes, leveraging the OS page cache, employing producer and consumer batching with configurable parameters, and utilizing zero‑copy sendfile to minimize CPU and memory overhead, enabling stable million‑message per second rates.
The article explains how storage media speed, kernel and user mode separation, and context‑switch overhead affect I/O performance, then details DMA, zero‑copy methods such as mmap + write and sendfile, the role of PageCache, and best practices like async and direct I/O for large‑file transfers.
This article explains how Apache Kafka attains million‑level write throughput by sequentially appending messages to log files, leveraging operating‑system page cache and asynchronous disk flushing, and details the step‑by‑step data flow from producer to broker to storage.
This article explains how Kafka sustains massive traffic by writing logs sequentially to disk, leveraging the operating system’s page cache for fast in‑memory writes, employing zero‑copy techniques like sendfile to avoid user‑space copying, and batching messages to reduce network overhead, thereby delivering high‑throughput, low‑latency streaming.
This article explains Linux's memory reclamation mechanisms, covering the role of memory as the system's bloodstream, the three reclamation paths (fast, direct, kswapd), zone watermarks, page cache structures, reverse mapping, and how swap and compression are used to keep the system stable under memory pressure.
This article explains how Kafka achieves high concurrency through its distributed broker cluster, partitioned topics, sequential log writes, message compression, asynchronous producer mechanisms, and OS page‑cache techniques, illustrating the combined architectural and performance optimizations that enable massive throughput.
This article explains the Linux Page Cache mechanism, covering its core concepts, read/write workflows, data consistency, optimization strategies, real-world use cases, advanced topics, common misconceptions, and practical tips for improving system performance and resource management.
Kafka attains million‑level message throughput by writing logs sequentially to disk, leveraging OS page cache for batch flushing, sending messages in large batches, and employing zero‑copy techniques that move data directly from disk to network sockets, dramatically reducing I/O overhead and CPU usage.
The article provides an in‑depth technical analysis of Linux’s memory reclamation system, covering zones, watermarks, page flags, LRU lists, the shrink_zone workflow, kswapd, direct reclaim, OOM handling, page‑cache structures, and practical tuning tips for optimal performance.
The article examines a long‑standing hidden bug in the Linux kernel’s page‑cache Xarray that caused occasional data loss with Large Folio support, details its discovery and fix by the TencentOS team, and shows how consolidating multiple tree walks into a single walk in Linux 6.10 reduced latency and improved performance by about ten percent.
This article examines the conditions under which data may be lost when a Linux process crashes while writing a file, explaining page cache behavior, the roles of stdio versus system calls, dirty page handling, write‑back mechanisms, and strategies such as fflush, fsync, and direct I/O to ensure data integrity.
This article explains how Kafka attains extremely high throughput by using batch sending, message compression, sequential disk I/O, PageCache, zero‑copy transfers, and memory‑mapped index files, detailing the underlying code paths and trade‑offs that enable millions of messages per second.
This article explains how RocketMQ achieves high throughput by using sequential disk writes, OS page caching, asynchronous flushing, and zero‑copy techniques, detailing each mechanism and providing code examples. It also discusses the impact on disk seek time and data durability.
The article explains how Kafka attains ultra‑high write throughput by leveraging disk sequential writes, zero‑copy data transfer, operating‑system page cache, and memory‑mapped files, detailing each technique’s impact on latency, CPU usage, and overall performance.
This article explains the design and implementation of file readahead in the Linux 3.12 kernel, using sequential, random, and multithreaded read scenarios to show how the kernel initializes readahead windows, triggers synchronous and asynchronous prefetches, and updates the page cache.
Through detailed code examples and step‑by‑step analysis, this article explains how the Linux 3.12 kernel implements file system readahead, showing how synchronous and asynchronous prefetch windows are initialized, adjusted, and used to improve sequential read performance.
This article explains how Linux reports memory usage, covering the free command output, the evolution of buffer and page caches, detailed /proc/meminfo fields, shared memory accounting, kernel memory statistics, huge pages, and per‑process metrics such as smaps and top.
Linux’s shmem subsystem provides hybrid anonymous/file‑backed pages that enable diverse shared‑memory scenarios—parent‑child communication, IPC, tmpfs, Android ashmem, and memfd—by using APIs such as shmem_file_setup, handling page faults through cache and swap mechanisms, and employing a specialized reclamation process to manage memory efficiently.
This article explains how Kafka’s design uses the operating system’s page cache and sequential disk writes to achieve fast, reliable persistence, debunking the myth that disk I/O is inherently slow and outlining practical configuration tips.
The article explains why the naïve approach of reading a file into a small user‑space buffer and sending it piece‑by‑piece is inefficient, and how zero‑copy, PageCache, asynchronous I/O and direct I/O can dramatically reduce context switches, memory copies, and CPU usage for high‑throughput file transfers.
This article explains why a naïve 32 KB‑chunk file transfer incurs excessive context switches and memory copies, and how zero‑copy, PageCache, asynchronous I/O, and direct I/O techniques dramatically reduce overhead and boost throughput for large‑scale data transfers.
This article examines traditional file transfer methods, highlights their performance drawbacks caused by excessive context switches and memory copies, and explains how zero‑copy, PageCache, asynchronous I/O, and direct I/O can dramatically improve throughput and reduce CPU usage in backend systems.
File transfer can be dramatically accelerated by reducing context switches and memory copies, using techniques such as zero‑copy, leveraging PageCache, and employing asynchronous or direct I/O, which together cut system calls, lower CPU usage, and improve concurrency for large‑scale data delivery.
Linux kernel introduced the struct folio abstraction to replace ad‑hoc compound‑page tricks, giving a clear collection‑oriented representation for power‑of‑two page groups such as THP and HugeTLB, and providing dedicated APIs that eliminate naming confusion, unify reference handling, and make memory‑management code safer and easier to understand.
The article explains how different storage media affect I/O speed, describes kernel and user mode separation, introduces DMA and zero‑copy techniques such as mmap + write and sendfile, and discusses PageCache behavior, advantages, drawbacks, and tuning for high‑performance file transfers.
The article explains the traditional Linux read/write system‑call I/O path, detailing the multiple CPU and DMA copies, context switches, page‑cache mechanisms, and advanced techniques such as zero‑copy, multiplexing, and direct I/O for performance optimization.
The article explains Linux I/O optimization through zero‑copy techniques—such as mmap + write, sendfile, and splice—detailing memory hierarchy, the benefits of reducing user‑kernel copies, the suitability of async + direct I/O for large file transfers, real‑world uses like Kafka and Nginx, and inherent platform limitations.
This article explains how Linux’s free command reports memory usage, clarifies the roles of buffer and page caches, demonstrates which caches can be reclaimed, and shows why tmpfs, shared memory, and MAP_SHARED mmap allocations remain in cache until explicitly released.
This article provides a comprehensive analysis of how JDK NIO performs file read and write operations by examining the underlying Linux kernel mechanisms, including the differences between Buffered and Direct IO, the structure and management of the page cache, file readahead algorithms, and the kernel parameters governing dirty page writeback.
This article explains the design background, performance benefits, potential drawbacks, synchronous and asynchronous mechanisms, key data structures, operational principles, illustrative examples, and critical code paths of Linux kernel file readahead, providing a comprehensive technical overview for developers and system engineers.
This article deeply explores the Linux kernel's handling of JDK NIO file reads and writes, detailing the role of page cache, radix trees, buffered versus direct I/O, prefetch algorithms, dirty‑page write‑back mechanisms, and the key sysctl parameters that control performance and data safety.
This article deeply explores JDK NIO's file reading and writing mechanisms, revealing how Buffered and Direct IO interact with Linux's page cache, the kernel's radix tree, prefetch algorithms, and dirty page management, while providing practical code examples and performance insights for developers.
This article explains the four key techniques—page‑cache usage, sequential disk writes, zero‑copy transfers, and partitioned segment indexing—that enable Kafka to achieve exceptionally high write performance, detailing how each mechanism reduces latency and maximizes throughput.
This article explains Linux storage hierarchy, the roles of user‑space and kernel caches, the three‑layer I/O stack, page‑cache synchronization policies, file‑operation atomicity, locking mechanisms, and performance testing techniques for HDD and SSD devices.
The article explains how to redesign a high‑traffic user‑registration flow by introducing an asynchronous queue layer, detailing the broker architecture, commitlog, consumeQueue, page‑cache, mmap, topic/tag routing, high‑availability strategies, and the role of a nameserver in a RocketMQ‑style system.
This article explains the Linux file I/O stack by outlining its clear route from user space to hardware, detailing the roles of VFS, the filesystem, the block layer, and the SCSI driver, and then dives into page cache mechanisms and code paths for both buffered and direct I/O.
This article explains the principles of disk I/O, covering read/write workflows, DMA acceleration, page cache, memory‑mapped files, buffering techniques, Linux kernel parameters and practical Java code examples to illustrate how to reduce CPU involvement and improve overall system performance.
The article explains the fundamentals of disk I/O, covering read/write processes, IO interrupts, DMA, page cache, mmap, buffered versus unbuffered file operations, ByteBuffer usage, Linux dirty‑page parameters, and how these mechanisms affect application performance and reliability.
This article explains how Linux implements traditional system‑call I/O using read() and write(), details the data‑copy and context‑switch overhead of read and write operations, describes network and disk I/O, and introduces high‑performance techniques such as zero‑copy, multiplexing, and page‑cache optimizations.
This article provides an in‑depth overview of Linux I/O mechanisms, covering the memory hierarchy pyramid, kernel‑level caching layers, the Linux I/O stack, page‑cache synchronization strategies, file‑operation atomicity, locking, and practical disk performance testing techniques.
Traditional Linux I/O relies on read() and write() system calls that involve multiple CPU and DMA copies and context switches, while modern optimizations such as zero‑copy, multiplexing, and page cache techniques reduce copying overhead, and understanding buffered, mmap, and direct I/O reveals their impact on performance.
This article explains the challenges of uncontrolled page‑cache growth in containerized workloads, reviews community attempts to limit it, and details TencentOS “Ruyi” memory‑QoS solutions—including cgroup‑level page‑cache limits, implementation details, and observed performance effects.
This article explains how Kafka achieves remarkable speed and massive throughput by using sequential disk I/O, OS page cache, zero‑copy transfers, partitioned log segments with indexes, batch processing, and efficient compression, making it a cornerstone of modern big‑data pipelines.
This article demystifies the Linux I/O stack by tracing a single‑byte read through system calls, VFS, page cache, filesystem layers, block management and I/O scheduling, revealing when real disk I/O occurs and how much data is actually transferred.
This article explains how a seemingly simple read‑of‑one‑byte in user code triggers a complex Linux I/O stack involving the I/O engine, system calls, VFS, page cache, file system implementations, generic block layer and I/O scheduler, and clarifies when actual disk I/O occurs and its granularity.
The article explains Linux page cache fundamentals, shows how to inspect and reclaim cache, and provides detailed tuning of vm.dirty_* and vm.swappiness parameters to smooth Kafka write traffic, reduce I/O spikes, and improve overall performance, illustrated with before‑and‑after benchmarks.
The article explains how Kafka achieves high throughput by using partition‑level parallelism, sequential disk writes with segment files, extensive use of the OS page cache, zero‑copy data paths, request batching and optional compression, while also discussing the underlying disk I/O principles.
This article explains how a simple read of a single byte in user space triggers a complex Linux I/O stack involving the read system call, VFS, page cache, generic block layer, and I/O scheduler, and clarifies when actual disk I/O occurs and how many bytes are transferred.
The Linux kernel’s page‑cache indexing migrated from the classic radix‑tree structure—using shift‑based multi‑way nodes, RCU‑protected insert, lookup, and delete operations—to the newer XArray API introduced in 4.20, which retains the radix layout while offering automatic resizing, built‑in locking, richer marking, and a cleaner, safer interface.
This article explains the hierarchical storage pyramid, Linux kernel I/O stack, page‑cache synchronization policies, the atomicity of write operations, the role of mmap and Direct I/O, and how disk characteristics influence multithreaded read/write performance and design decisions.
This article explains why Kafka achieves remarkable speed by relying on Linux page cache, detailing the differences between page and buffer caches, Kafka's zero‑copy I/O path, relevant kernel parameters, and tuning recommendations for optimal backend performance.
This article explains how memory-mapped files are used in a search engine, describes the underlying Linux mechanisms such as page cache, multi-level page tables and radix trees, and presents practical methods—including mincore, /proc smaps, pagemap and system commands—to analyze and monitor their memory consumption at both kernel and process levels.
Kafka achieves its ultra‑high throughput and low latency by writing data to the OS page cache, performing sequential disk writes, and employing zero‑copy techniques that eliminate unnecessary data copies during consumption, enabling tens of thousands to millions of messages per second.
The article explains how Kafka achieves high throughput and low latency by leveraging sequential disk I/O, operating‑system page cache, zero‑copy transmission, and a partition‑segment storage model, all of which are key design choices for big‑data messaging systems.
This article explains how disk I/O performance is shaped by page cache behavior, sequential versus random access, elevator scheduling, and storage data structures such as B+Tree and LSM trees, showing why operating systems and databases must design for sequential reads and writes to achieve optimal throughput.
This article explains Linux kernel memory management, covering process address space allocation, OOM selection criteria, memory mapping types, cache behavior, tmpfs, and both manual and automatic memory reclamation mechanisms.
This article explains how the Linux free command reports memory usage, clarifies the roles of buffer and page cache, shows how to manually drop caches, and reveals scenarios—such as tmpfs, shared memory, and mmap—where cached memory cannot be reclaimed, helping readers achieve a deeper understanding of system memory behavior.
This article explains how the Linux free command reports memory usage, clarifies the roles of buffer cache and page cache, demonstrates which cache memory can be reclaimed, and shows practical tests using tmpfs, shared memory, and mmap with code examples.
