Master Disk I/O Performance Tuning on Kunpeng Processors: Tools & Tips

1. Tuning Overview

The Special Forces team was formed at the end of June and has been operating for over a hundred days; this series aims to help developers working on Kunpeng processors with software development and performance optimization. This article focuses on disk I/O subsystem performance tuning.

1.1 Tuning Idea

CPU cache, memory, and disk have vastly different access speeds. When required data is not in cache or memory, the CPU stalls waiting for disk reads. Modern systems use multi‑level storage (cache, RAM, SSD, HDD) and various scheduling algorithms to mitigate this mismatch. By leveraging data locality, hot data can be prefetched from disk to reduce CPU wait time, and many optimizations revolve around better cache utilization for improved I/O performance.

File‑system‑level optimizations are also covered.

1.2 Key Tuning Parameters

2. Common Performance Monitoring Tools

iostat

Introduction

iostat is the most frequently used tool for investigating disk I/O problems. It aggregates online disk statistics, providing metrics for load characteristics, utilization, and saturation. It can be run by any user, and its overhead is negligible because the data comes directly from the kernel.

Installation

iostat is usually installed with the system. On CentOS, install it with: # yum -y install sysstat Usage

Run the command with desired options, for example: # iostat -d -k -x 1 100 This collects statistics every second for a total of 100 seconds.

Common Parameters

The output includes fields such as rrqm/s, wrqm/s, %util, await, svctm, and avgqu-sz. Important interpretations: rrqm/s and wrqm/s show the number of merged read/write requests per second; non‑zero values indicate sequential I/O.

If %util approaches 100 %, the I/O subsystem is saturated, leading to higher await and increased CPU wait time. await is the average wait time for I/O requests; compare it with svctm to determine whether delays are due to queueing or device latency. avgqu-sz reflects the average queue length and should be observed over a longer period.

3. Optimization Methods

3.1 Adjust Dirty Data Flush Strategy

Principle

Dirty pages in the page cache must be written back to disk. Applications can write directly to disk (O_DIRECT) or let the page cache handle writes, reducing disk I/O.

Parameters /proc/sys/vm/dirty_expire_centisecs – time (in centiseconds) a dirty page can stay in cache before being flushed. Default is 3000 (30 s). Reduce to spread writes for sequential workloads.

# echo 2000 > /proc/sys/vm/dirty_expire_centisecs

/proc/sys/vm/dirty_background_ratio

– percentage of memory that can be occupied by dirty pages before the background flusher starts. Default 10. Increase to allocate more write buffer, or decrease for write‑intensive workloads.

# echo 8 > /proc/sys/vm/dirty_background_ratio

/proc/sys/vm/dirty_ratio

– maximum percentage of memory that can be dirty. Default 40. Raising this value can delay synchronous writes for write‑heavy applications.

Note: Larger dirty caches increase risk of data loss on power failure; critical data should use O_DIRECT.

3.2 Tune Disk Read‑Ahead Parameters

Principle

Read‑ahead prefetches adjacent blocks based on locality. Increasing the value helps sequential reads but wastes bandwidth for random access.

Adjustment

Modify read_ahead_kb for the target block device, e.g., on CentOS: # find / -name read_ahead_kb Default is 128 KB. To set 4096 KB: # echo 4096 > /sys/block/$DEVICE_NAME/queue/read_ahead_kb Adjust according to workload characteristics.

3.3 Optimize I/O Scheduler

Principle

The kernel I/O scheduler can reorder and merge requests to improve performance. Three common schedulers:

CFQ – Completely Fair Queue, default in older kernels; fair but not optimal for heavy I/O.

Deadline – maintains separate read/write queues with timeouts; good for high‑load, HDD‑heavy scenarios.

NOOP – simple FIFO; best for SSDs where the device handles ordering.

Change Scheduler

Check current scheduler: # cat /sys/block/$DEVICE_NAME/queue/scheduler Modify, e.g., set deadline for sda:

# echo deadline > /sys/block/sda/queue/scheduler

3.4 File System Parameter Tuning

Choosing a high‑performance file system (e.g., XFS) and mounting with appropriate options can improve throughput.

Disable Barrier

Barrier ensures journal consistency but adds latency. On systems with protected RAID or battery‑backed caches, mount with nobarrier: # mount -o nobarrier -o remount /home/disk0 Note: Disabling barrier risks data loss on sudden power loss unless hardware provides protection.

Use XFS

XFS offers high scalability and is well‑suited for large files. When creating XFS, you can specify a larger block size for better performance:

# mkfs.xfs /dev/sda1

# mkfs.xfs /dev/sda1 -b size=8192

3.5 Use Asynchronous I/O (libaio)

Principle

Synchronous file reads block the thread. libaio provides asynchronous I/O, allowing a single thread to handle multiple read/write requests without blocking, especially when combined with epoll.

Implementation details can be found in the linked Huawei Cloud forum post.

https://bbs.huaweicloud.com/forum/thread-27343-1-1.html