How to Slash Linux Disk Fragmentation and Boost I/O Performance by Up to 5×

Optimizing Under Existing Filesystems

The Linux kernel and various filesystems provide several optimizations to speed up disk access, but these must be coordinated with server design to be effective.

Filesystem Cache

Linux allocates most free memory to the virtual file system as a page cache, which is evicted using an LRU algorithm when memory runs low. The free command shows the cached portion. By analyzing access patterns and implementing a custom eviction algorithm, cache hit rates can be dramatically increased; for an HTTP forward proxy, a well‑tuned algorithm can achieve the effect of a 100 GB LRU cache with only 1 GB of RAM. If you do not write a new algorithm, an additional user‑space file‑cache program is usually unnecessary.

Minimum Allocation

Setting a larger minimum allocation can waste disk space, especially when many small files exist. The allocation size should be chosen based on the workload; changing it often requires source‑code modifications.

I/O Scheduling

I/O scheduling can greatly improve performance if the application issues enough concurrent I/O requests. To issue multiple I/O operations from user space, use asynchronous calls such as aio_read.

Tip: Making a file descriptor non‑blocking does not prevent the process from sleeping on disk I/O. Under normal conditions a read may block for a few milliseconds, but under heavy I/O it can cause sleeps of ten seconds or more. The kernel function do_generic_file_read calls lock_page_killable, which enters sleep without checking the non‑blocking flag.

Read‑Ahead

The Linux kernel can predict future read requests and prefetch data, reducing the number of I/O operations and lowering latency.

Delayed Allocation

When a file grows, allocation can be delayed and aggregated in memory before a single large disk write. Side effects include: (1) if the application calls fsync after each write, delayed allocation is ineffective; (2) occasional large I/O latency may occur because many blocks are written at once.

For more details, see http://jsmylinux.no-ip.org/applications/using-e4defrag/ .

Storing frequently accessed images contiguously can boost I/O performance up to tenfold. Traditional sprite‑sheet techniques merge images into one large file, whereas using e4defrag to keep related files in the same directory and periodically defragment them achieves similar benefits without extra front‑end processing.

Implementing a Custom Filesystem

We built a dedicated filesystem for proxy servers that increased disk I/O performance by 3–5×. Most servers do not need to support file modification; files are created once and then only read or deleted. This constraint eliminates fragmentation, allowing the disk to operate at near‑theoretical limits.

Files larger than 16 MB are allocated in 16 MB chunks; subsequent expansions also use 16 MB increments. Small files experience zero fragmentation, requiring only a single seek. The efficiency of disk I/O can be expressed as:

Disk Utilization = Transfer Time / (Average Seek Time + Transfer Time)

On a 1 TB 7200 RPM SATA drive, sequential 16 MB reads/writes achieve over 98 % utilization.

Click the lower left corner for more details