How to Implement Storage Tiering on Linux: Block Device vs LVM Solutions

Background Introduction

With the advancement of SSD and SAS technologies, storage media vary widely in performance and cost. SSDs offer much higher random I/O speed than traditional HDDs but are more expensive per gigabyte, prompting the need for efficient tiered storage.

Tiered storage (Storage Tiering) automatically places hot data on high‑performance devices and cold data on slower devices, balancing performance and cost. While many enterprise arrays implement this in firmware, the article explores how to achieve similar functionality on a Linux host.

Key Technical Points of Tiered Storage

Tiered storage requires virtualizing multiple physical devices (DEV1, DEV2) into a single virtual device (VDEV). All I/O is directed to VDEV, which maps requests to the appropriate physical device based on address mapping. The system must also monitor I/O hotness and migrate data between tiers.

Virtualization of storage devices

I/O performance monitoring

Data migration based on hotness

Block‑Device‑Based Tiered Storage Solution

Architecture

The solution consists of a Linux kernel driver and several user‑space control programs. The driver handles virtualization, I/O monitoring, and data migration; the user‑space tools create and delete virtual devices, trigger migrations, and query status.

Implementation Details

Device registration via register_chrdev or misc_register and IOCTL interface.

Creation of a virtual block device (VDEV) using register_blkdev.

Mapping table between VDEV blocks and physical device blocks, stored on disk for persistence.

I/O hotness statistics collected per block, ignoring large sequential I/O.

Automatic or manual data migration driven by a kernel timer or user command.

Open‑Source Implementation

BTier implements the above design, supporting up to 16 devices. After building and loading the module, a character device tiercontrol appears, and the btier_setup utility can create a btiera block device for normal use. BTier exposes sysfs interfaces for migration control and statistics.

LVM‑Based Tiered Storage Solution

LVM provides storage virtualization and built‑in data‑migration APIs, making a tiered solution simpler.

Architecture

The design relies on LVM’s volume groups and logical volumes, with hotness analysis performed in user space and data movement handled by pvmove.

Implementation Details

Create physical volumes (PV) and a volume group (VG), then a logical volume (LV) that spans the devices.

Collect I/O hotness using blktrace and blkparse or a custom driver.

Use pvmove to migrate hot data to SSDs and cold data to HDDs.

Open‑Source Implementation

LVMTS (LVM Tiered Storage) implements this approach with daemon processes that monitor blktrace output, record statistics, and invoke pvmove according to configuration.

Conclusion

Linux offers multiple ways to realize storage tiering, such as a kernel‑level block‑device driver or an LVM‑based user‑space solution. Each has trade‑offs in complexity, flexibility, and performance, and further considerations include thin provisioning and SSD TRIM support.