Building a Scalable Linux Storage Stack: RAID, LVM, XFS & Monitoring from Scratch

Problem Background: Most Storage Incidents Stem from Poor Planning

Common failures include root partition filling while data disks have free space, inability to expand filesystems without downtime, hot data stuck on slow disks, and inconsistent filesystem layouts across servers. The root cause: storage lacked layered design from day one.

Goals & Scope

Online expansion : add disks and grow filesystems without data migration or downtime.

Single-disk failure tolerance : automatic degradation and rebuild on disk replacement.

Proactive disk health alerts : SMART monitoring via smartd with custom notifications.

Workload isolation : separate logical volumes for data, logs, and temp to prevent cross-impact.

Rollback capability : LVM snapshots and metadata archives before every change.

Applicable to physical or self-managed VMs hosting database replicas, log archives, backup storage, object storage nodes, or container host data directories with two or more blank block devices. Not needed where cloud-native block storage (with snapshots) or distributed storage (Ceph) already exists.

Core Concepts & Selection

Block Device Layer Stack

Application write
 |
 Filesystem (XFS / ext4)          ← manages files & directories
 |
 Logical Volume (LVM)             ← capacity pooling, online resize, snapshots
 |
 RAID (mdadm soft RAID or HW)     ← single-disk fault tolerance
 |
 Physical block devices (SATA/SAS/NVMe) ← final data persistence

Each layer has distinct responsibilities; never let one layer encroach on another's duty.

Partitioning: GPT vs Whole Disk

Disks >2 TiB require GPT (use gdisk or parted). Two approaches: (1) create a GPT partition of type lvm or fd per disk — easier identification via blkid and more stable with RAID controllers; (2) use whole disk directly ( pvcreate /dev/sdb) — simpler but less clear boundaries. Production commonly uses partitions for RAID members or LVM PVs.

RAID Level Selection

Decision matrix based on workload:

Random write heavy (DB, VM images, log aggregators) : RAID10 (near=2 layout). 4-disk RAID10 loses 50% capacity but delivers best write performance and per-pair fault tolerance.

Sequential write heavy (backup, video streams) : RAID6 for higher capacity density; accept write penalty if workload permits.

Small critical datasets : RAID1 mirror, simplest.

Hardware RAID present : let the card handle RAID; OS sees a single block device. Never stack mdadm on top of a hardware RAID volume.

LVM Benefits

Space pooling : multiple RAID devices form one VG; a single filesystem spans many disks. Expansion adds new PV to VG — no data migration.

Online changes : lvextend -r grows LV and filesystem (XFS via xfs_growfs) while mounted.

Snapshots : LVM snapshots with fsfreeze or application quiesce provide point-in-time rollback anchors.

XFS vs ext4

XFS is default on RHEL 7+, excels at large files/high-concurrency sequential writes and large directories (delayed allocation). Both support online growth; only ext4 supports shrinking (offline). Rule: prefer XFS for data volumes ; choose ext4 only if future shrinkage is certain (e.g., temporary volumes).

Thin Provisioning (thin pool)

Create a thin pool in the VG; logical volumes allocate virtual space, consuming physical space only as written. Ideal for container hosts, VM backends, or workloads that request more than they use. Risk : overcommitment. Must configure thin_pool_autoextend_threshold=80 and thin_pool_autoextend_percent=20 in /etc/lvm/lvm.conf and monitor pool usage ( data%) with alerts at 80%. If VG free space exhausts, auto-extend fails and writes return I/O errors.

Environment & Pre-flight Checks

Hardware: 1×480 GB SATA SSD (system), 6×4 TB SAS HDDs (sdb–sdg, JBOD), 2×4 TB SAS (sdh/sdi for expansion demo). OS: Rocky Linux 9.8. Install

lvm2 xfsprogs mdadm smartmontools nvme-cli fio sysstat parted

.

Critical pre-checks (skip = gambling): lsblk to confirm target disks exist and are unmounted. blkid on each target — must show no signatures. mdadm --examine — must report no md superblock. smartctl -H and smartctl -l error — health OK, no media errors. pvs; vgs; findmnt /data — ensure no existing LVM or mounts on targets.

High-risk commands ( wipefs -a, mdadm --zero-superblock, pvcreate) are irreversible. Verify checks 2 & 3 are clean before proceeding.

Step 1: mdadm Software RAID10

Create Array

mdadm --create /dev/md10 
  --level=10 --raid-devices=6 --bitmap=internal 
  /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg

--level=10

: RAID10, default near=2 layout (mirror on adjacent disk), superior 4K random write vs RAID5/6. --bitmap=internal: internal write-intent bitmap accelerates rebuild by resyncing only dirty chunks (5–10% write overhead; omit for write-heavy log volumes).

Members are whole disks (JBOD direct). If using partition scheme, create one GPT fd partition per disk first.

Never use --assume-clean on used disks — it skips initial sync and risks silent data corruption.

Monitor sync via watch -n 2 'cat /proc/mdstat'; wait for finish=0.0min and no resync line.

Persist Configuration

mdadm --detail --scan | tee -a /etc/mdadm.conf
dracut -f

Verify: grep -c 'ARRAY /dev/md10' /etc/mdadm.conf returns 1; mdmonitor.service enabled.

Array Monitoring

mdmonitor

runs mdadm --monitor --scan, logs events to journal. Integrate with internal alerting (Chapter 8 script).

Step 2: LVM & Filesystem

PV → VG → LV

pvcreate /dev/md10
vgcreate data_vg /dev/md10
lvcreate -n lv_app -L 7t data_vg
lvcreate --type thin-pool -L 3t -n pool_volumes data_vg

Capacity accounting: 6×3.6 TB RAID10 ≈ 10.9 TB usable. lv_app 7 TB + pool_volumes 3 TB = 10 TB, leaving ~0.9 TB headroom for thin auto-extend and safety. Never fill VG completely. Design: lv_app for predictable workloads (DB, logs); thin pool for overcommitted workloads (containers, VMs).

XFS Formatting & Alignment

Let mkfs.xfs auto-detect RAID topology via lsblk -t (shows OPT-IO, MIN-IO). Do not manually specify -d su=,sw= unless you fully understand the chunk layout; modern xfsprogs handles alignment correctly.

mkfs.xfs -L data-lv-app /dev/data_vg/lv_app
xfs_admin -u /dev/data_vg/lv_app  # record UUID for fstab

Mount Configuration

mkdir -p /data
cat >> /etc/fstab << 'EOF'
UUID=1a8f3b9e-4c2d-4b6f-9e5d-7a8b1c2f3d4e  /data  xfs  defaults  0 0
EOF
mount -a && df -h /data

Mount options: critical data (DB, production) — no nofail (enter emergency on failure); logs/cache/rebuildable data — add nofail.

Thin Pool Guardrails

In /etc/lvm/lvm.conf:

activation {
  thin_pool_autoextend_threshold = 80
  thin_pool_autoextend_percent = 20
}

Activate: lvchange --refresh data_vg/pool_volumes. Verify systemctl is-active dmeventd and lvs -o name,attr shows twi-a-tz-- (thin pool monitored). Second layer: alert on data% > 80 via monitoring (Chapter 8).

Periodic fstrim

fstrim.timer

(weekly, default enabled) discards unused blocks back to thin pool. Verify with fstrim -v /data. Do not run during peak I/O — discard causes latency spikes on RAID.

Online Expansion Drills

Scenario A: Add Two New Disks (sdh, sdi)

Pre-check: blkid, mdadm --examine, smartctl -H on new disks.

Create RAID1 pair:

mdadm --create /dev/md11 --level=1 --raid-devices=2 /dev/sdh /dev/sdi

.

Add to VG: pvcreate /dev/md11 && vgextend data_vg /dev/md11.

Grow LV & FS: lvextend -L +3.5t -r data_vg/lv_app (single atomic step).

Verify: lvs shows larger LSize; df -h /data shows grown Size with unchanged Used.

Scenario B: Cloud/SAN Disk Resize (Same Disk Grows)

echo 1 > /sys/block/sdh/device/rescan   # or nvme ns-rescan
parted -s /dev/sdh resizepart 1 100%
pvresize /dev/sdh1
lvextend -l +100%FREE -r data_vg/lv_app

Confirm device identity with lsblk / blkid before resizepart.

Why Not mdadm --grow for RAID10?

RAID10 reshape reshuffles data across disks, causing unpredictable write performance degradation and complex recovery on failure. Adding a new RAID array and vgextend leaves existing data untouched — risk and duration are controllable. Scalability comes from VG multi-PV evolution, not array reshape.

Rollback for Expansion

vgcfgrestore --list data_vg

lists archived metadata snapshots in /etc/lvm/archive. To revert:

vgchange -an data_vg && vgcfgrestore -f /etc/lvm/archive/data_vg-<timestamp>.vg data_vg && vgchange -ay data_vg

. Note: only metadata rolls back; XFS cannot shrink online — shrinking requires data copy, rebuild, copy back. Plan capacity ceilings upfront.

Monitoring & Alerting

smartd Health Patrol

/etc/smartmontools/smartd.conf

:

DEVICESCAN -H -l error -l selftest -f 
  -W 4,45,55 
  -n standby,10,q 
  -s (S/../.././02|L/../../6/03) 
  -m root -M exec /usr/local/bin/smartd-notify.sh

-W 4,45,55

: warn at 45°C, critical at 55°C. -s: short test daily 02:00, long test Saturday 03:00. -n standby,10,q: skip standby disks (max 10 consecutive skips), quiet. -M exec: hand off to internal notification script (WeCom/Feishu/Slack).

Restart smartd after config change. Test: smartd -q onecheck.

Notification Script

/usr/local/bin/smartd-notify.sh

(0755) reads token/URL from /etc/sysconfig/smartd-notify (root:root 0600), logs to journal via logger, optionally POSTs to webhook. Test: /usr/local/bin/smartd-notify.sh EmailTest /dev/sdb.

node_exporter Metrics (PromQL)

# Capacity alert (threshold from business baseline)
node_filesystem_avail_bytes{mountpoint="/data"} / node_filesystem_size_bytes{mountpoint="/data"} * 100 < 15

# Inode alert (critical for many small files)
node_filesystem_files_free{mountpoint="/data"} / node_filesystem_files{mountpoint="/data"} * 100 < 10

# MD degraded
node_md_state{state="degraded"} == 1

# Write latency (w_await equivalent)
rate(node_disk_write_time_seconds_total{device=~"sd[b-g]"}[5m])
  / rate(node_disk_writes_completed_total{device=~"sd[b-g]"}[5m]) > 0.02

Thresholds must derive from business baselines, not copied defaults.

iostat Baseline Capture

iostat -dx 1 5 /dev/sd{b,c,d,e,f,g}

records normal %util, await, w_await. High %util + rising await = hotspot; high %util + stable await = busy but healthy. Store baseline in runbook.

Performance Validation: fio

High-risk — only on fresh environments, dedicated test volumes, or under LVM snapshot with maintenance window.

# 4K random write (DB baseline)
fio --name=randw4k --filename=/data/fio-tmp/rw4k 
  --size=8g --rw=randwrite --bs=4k --direct=1 --iodepth=32 
  --numjobs=1 --runtime=60 --time_based --group_reporting

# 1M sequential write (backup baseline)
fio --name=seq1m --filename=/data/fio-tmp/seq1m 
  --size=8g --rw=write --bs=1m --direct=1 --iodepth=8 --numjobs=1 
  --runtime=60 --time_based --group_reporting

# Cleanup
rm -f /data/fio-tmp/* && rmdir /data/fio-tmp && sync

Pass criteria: throughput/IOPS within 20% of comparable baseline. >50% gap indicates array/topology issue — revisit alignment (Section 6.2), not application code.

Troubleshooting Loop: End-to-End Drill

Case : Nightly batch window write latency spike alert ( w_await > 20ms), /dev/sdd at 42ms for 5 min.

Phenomenon : capacity 14%, inodes 3%, not network.

Initial hypothesis : single disk stalling RAID write rhythm.

Check : iostat -dx shows

sdd

w_await 38ms

, %util 98% vs peers 4–7ms, 60–80%. dmesg -T | grep sdd reveals Medium Error on sector 1953125000. smartctl -l error /dev/sdd confirms. mdadm --detail /dev/md10 shows State: clean, degraded, Failed Devices: 1, sdd(F).

Root cause : media damage on sdd; RAID10 near=2 degraded but data intact (other mirror alive). Repair window must shrink — second paired disk failure = data loss.

Repair : mdadm /dev/md10 --fail /dev/sdd --remove /dev/sdd; physically replace disk (identify via /dev/disk/by-id); verify new disk clean ( blkid); mdadm /dev/md10 --add /dev/sdd; watch /proc/mdstat for rebuild.

Verify : rebuild finishes (all U, state clean); new disk SMART OK; iostat latency returns to baseline for 24h.

Rollback plan : throttle rebuild speed via sysctl dev.raid.speed_limit_min/max; never pause rebuild. If new disk also errors, stop — escalate to hardware/slot investigation. Dual simultaneous failure: restore from latest LVM snapshot + cold backup.

Postmortem : smartd logged Medium Error earlier but alert not routed; add direct log-alert on Medium Error keyword; stock cold spare; update runbook.

Rollback Handbook

fstab Error → Emergency Shell

Identify bad line: cat /etc/fstab | grep -v '^#'.

Get real UUID: xfs_admin -u /dev/data_vg/lv_app.

Fix fstab, then mount -a; findmnt /data.

Prevention: after fstab edit, run findmnt --verify -v /data; non-critical volumes use nofail.

LVM Metadata Mistake (e.g., wrong vgextend )

vgcfgrestore --list data_vg
vgchange -an data_vg
vgcfgrestore -f /etc/lvm/archive/data_vg-<timestamp>.vg data_vg
vgchange -ay data_vg

Accidental mdadm --remove on Active Disk

Single disk: immediately mdadm /dev/md10 --add /dev/sdd — rebuild restores redundancy.

Multiple disks + --zero-superblock: check mdadm --examine first; if superblocks gone, recovery complexity spikes. Business must authorize; restore from cold backup — no live experiments.

Degraded Operation Limits

RAID10 degraded is acceptable short-term, but replace disk immediately if: (1) second disk shows media error/SMART warning; (2) rebuild speed stays low (monitor via node_exporter); (3) HW RAID battery/cache state changes.

Production Standards: Scripts, Change Control & Audit Trail

Data Center Pre-flight Script

disk_preflight.sh

(read-only by default; --trim-run enables interactive fstrim). Features: mapfile for safe disk listing, tee -a for local logs, set -euo pipefail, read -r for human confirmation on destructive actions, credentials via env file, script body read-only.

Permissions & Change Discipline

Least privilege : high-risk commands ( wipefs -a, mdadm --zero-superblock, pvcreate, lvremove) in sudo whitelist per person+scenario with audit logs.

Windows & canary : validate full flow in staging before production; rebuild, trim, fio never overlap business peaks.

Two-person review : array changes require two named engineers + ticket number.

Traceability : all commands and script output to /var/log/storage_build.log; LVM metadata archives in /etc/lvm/archive; cross-check both at start and end of work.

Summary

A production-grade Linux storage environment is not six commands ( partition → mkfs). It is the combination of four disciplines:

Layered design : mdadm owns RAID, LVM owns volumes, XFS owns filesystem. vgextend + lvextend -r becomes muscle memory.

XFS only grows : plan each volume's ceiling; uncertain workloads go to thin pool with dual guardrails (auto-extend + alerting).

Know before it fails : smartd + notification script + node_exporter (capacity/inode/latency/MD state) — thresholds from your baselines.

Every change has an exit : fstab verification, LVM metadata archives, RAID rebuild throttling, LVM snapshots with fsfreeze as the safest rollback anchor in change windows.

All write operations via script : scripts enforce dry-run/confirmation, emit logs — both engineering practice and first evidence in postmortems.