Production LVM + XFS Expansion: Complete Guide from PE Granularity to AG Alignment
This article provides a comprehensive guide to expanding LVM + XFS filesystems in production, covering the four-layer address space alignment, PE and AG granularity, multipath considerations, step-by-step commands, verification, monitoring, and troubleshooting common failures.
Introduction: Why Expansion Fails in Production
LVM + XFS expansion appears to be a simple four-step pipeline: add disk → pvcreate → vgextend → lvextend -r. In test environments it always succeeds, but in production it can fail in seven or eight different ways, all showing the same symptom: df numbers do not change.
To understand why, we must examine the four independent address spaces involved:
Physical Layer Logical Layer Block Device Layer Filesystem Layer
+---------+ +-----------+ +----------+ +-----------+
| /dev/sdb| ---> | VG data | ---> | /dev/dm-3| ---> | XFS |
| 1.8 TiB | | PE 4 MiB | | LV size | | AG split |
| 512B sec| | extent int| | arbitrary| | bsize=4k |
+---------+ +-----------+ +----------+ +-----------+
^ ^ ^ ^
| | | |
pvcreate vgcreate lvextend xfs_growfs
PV label 4MiB PE=PV/PE size by LE alloc only grow
tail reserve ~1MiB cannot exceed PV modify dm table grow by AGEach layer has its own constraints, and every expansion step translates the size from the upper layer into a form the lower layer can accept.
1.1 XFS Allocation Group (AG) Mechanism Determines Expansion Granularity
XFS divides the filesystem into Allocation Groups (AGs), each with its own free-space B+tree, inode B+tree, and metadata redundancy. This enables high-concurrency writes because different AGs can be locked independently. mkfs.xfs chooses agcount based on device size:
< 16 GiB : default agcount usually 4, AG size several GiB
16 GiB – 1 TiB : agcount = 4, AG size 4 – 256 GiB
1 TiB – 4 TiB : agcount = 8, AG size 128 – 512 GiB
> 4 TiB : agcount = 16, AG size 256 GiB+
When xfs_growfs expands, if the new space cannot form a complete new AG, it is appended to the last AG. This has two engineering consequences:
Small incremental expansions never increase AG count , so write concurrency does not improve. A 4-AG 4 TiB filesystem expanded ten times by 100 GiB each remains 4 AGs, losing real performance for high-concurrency database workloads.
xfs_growfs has a minimum increment floor . Although documentation says minimum is one block (bsize), the practical constraint comes from the need to read/write the last AG's free-space tree; too small an increment returns data size unchanged, skipping.
The correct approach is expand by AG granularity : calculate the size of one AG, then make lvextend increments an integer multiple of that value so each expansion adds a full AG and concurrency scales linearly.
1.2 Three Irreversible Red Lines
mkfs.xfs -foverwrites existing filesystem — the #1 cause of production accidents when a mounted LV is mistaken for a new disk. Hence all scripts enforce --dry-run. lvreduce on XFS is a dead end. XFS cannot shrink; after lvreduce filesystem metadata extends beyond the LV boundary, causing duplicate UUID or Structure needs cleaning on mount. No online recovery exists; only backup restore.
Changing pe_size equals rebuilding the VG. vgconvert -s 8m on older versions required all LVs offline; newer LVM simply refuses. PE size must be chosen once.
Design principle: every step verifiable, every command replayable, metadata backup before any irreversible action.
Baseline Environment
OS: Rocky Linux 9.4 / kernel 5.14.0-427.el9
LVM: lvm2 2.03.23 (devices file enabled)
FS: XFS (default crc=1 reflink=1 bigtime=1)
Disks: 2x 1.8 TiB SAS SSD (multipath, 2 paths)
Array: RAID 10, stripe unit 256 KiB, per-disk 1.8 TiB
Mount: /data (database & logs)
Current: VG data total 3.6 TiB, LV lvdata 2 TiB, free 1.6 TiB
Goal: Add 2 TiB from array, grow /data to 4 TiB
Constraint: Database online, no downtime; block-device changes only in maintenance windowPre-Expansion Health Checks
Most expansion accidents stem from pre-existing dirty state: leftover md signatures, multipath duplicate devices, pvcreate on both partition and whole disk, /etc/fstab using UUID while LVM metadata changed. Check first, act later.
2.1 Full Status Snapshot
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,ROTA
pvs -o pv_name,vg_name,pv_size,pv_free,pv_uuid
vgs -o vg_name,vg_size,vg_free,vg_extent_size,vg_extent_count
lvs -o lv_name,vg_name,lv_size,seg_count,devices
df -hT /dataFive critical confirmations from the output:
sdb/sdc and mpatha/mpathb coexist → multipath in use. Never run pvcreate on /dev/sdb (see §5.1).
lvdata segment spans two PVs, second PV has 1.60T free → linear mode (not striped). Adding disks will not improve write performance.
VG extent size 4.00m, #Ext 943373 → all -l calculations use 4 MiB units.
df shows 2.0T but Used+Avail = 2.012T → difference is XFS metadata/reserved blocks, normal.
/data mounted on rl-data, not data-lvdata → two different VGs on same machine; never mix lsblk outputs across hosts.
2.2 Six Hard Checks
/etc/fstab mount method : /data uses device path ( /dev/mapper/data-lvdata), not UUID. Correct for LVM (LV name stable), but if VG/LV renamed, fstab must be updated or boot enters emergency mode.
Multipath topology converged : multipath -ll must show two paths per mpath, both active ready running. failed faulty means a link is down — expansion may succeed but I/O falls to single path, halving performance and losing redundancy. Fix links first.
LVM device recognition : RHEL 9 uses devices file ( /etc/lvm/devices/system.devices), not lvm.conf filter. Conflict causes pvcreate to report Device is not in devices file or silently ignore. Run lvmdevices --check and inspect the file.
PV labels not duplicated by multipath :
pvs -a -o pv_name,pv_uuid,vg_name,pv_size | grep -E 'sd[b-z]|mpath'. /dev/sdb, /dev/sdc should show empty UUID and VG. If they have UUID/VG, duplicate PV exists — severe fault (§5.1).
Read-only XFS consistency check : xfs_repair -n /dev/mapper/data-lvdata (must be unmounted or mounted read-only with norecovery). Clean exit shows No modify flag set. Errors require downtime repair before expansion.
Backup LVM metadata :
vgcfgbackup -f /root/lvm-backup/data-$(date +%F-%H%M).vg dataand copy /etc/lvm/archive. Restore with vgcfgrestore only if metadata corrupted and no data written since backup.
2.3 Stripe & Alignment Pre-Check
Confirm whether LV is striped (
lvs -a -o name,vg_name,lv_size,stripes,stripe_size,devices data). Stripes=1 → linear. Converting to striped requires new LV + data migration (high risk, not in this expansion). If striped, xfs_info should show sunit / swidth matching array geometry (e.g., sunit=64 (256 KiB), swidth=128 (2 data disks)). sunit=0 means misaligned at creation; expansion won't fix it.
Core Six-Step Expansion Process
3.1 Step 0: Make New Disk Visible & Stable
for h in /sys/class/scsi_host/host*/scan; do echo '- - -' > "${h}" 2>/dev/null || true; done
sleep 5
lsblk -o NAME,SIZE,TYPE,TRAN | grep -E 'disk|mpath'Expect one new mpath device (e.g., mpathc), not two raw sd devices. If multipath doesn't aggregate, check /etc/multipath.conf blacklist.
3.2 Step 1: pvcreate & Lock to devices file
pvcreate --yes --labelsector 1 /dev/mapper/mpathc
pvs -o pv_name,pv_size,pv_free,pv_uuid /dev/mapper/mpathc
lvmdevices --check--labelsector 1writes PV label at sector 1 (for arrays using sector 0). Do not use on ordinary disks. If disk had previous VG or md, wipe first:
wipefs -a /dev/mapper/mpathc; mdadm --zero-superblock /dev/mapper/mpathc(only after confirming no data via lsblk -f, pvs, mdadm --detail).
3.3 Step 2: vgextend & Verify Allocatable Space
vgextend data /dev/mapper/mpathc
vgs -o vg_name,vg_size,vg_free,vg_extent_count,vg_extent_size dataVG free extents increase from 943373 to 1467392 (Δ524019 extents ≈ 2046.9 GiB), slightly less than nominal 2 TiB due to PV label reserve and PE rounding. df unchanged — filesystem unaware.
3.4 Step 3: Calculate Increment & lvextend
Three syntaxes:
A: by PE count (most precise) lvextend -l +512000 /dev/data/lvdata (2000 GiB = 512000 PE)
B: by %FREE (drain VG) lvextend -l +100%FREE /dev/data/lvdata C: by absolute size (intuitive, may round) lvextend -L +2000G /dev/data/lvdata Output shows changed from X (N extents) to Y (M extents); M-N is actual PE gain. Multiply by 4 MiB for real bytes.
AG-granularity calculation (baseline: agsize=268435455 blks, bsize=4096):
ag_bytes=$((268435455 * 4096))
pe_needed=$(( (ag_bytes + 4*1024*1024 - 1) / (4*1024*1024) ))
# pe_needed = 262144 extents = 1024 GiB per AGThus each 1024 GiB adds one AG. Available 2046 GiB → two full AGs (2048 GiB). If only 800 GiB added, AG count unchanged.
3.5 Step 4: xfs_growfs with Mount Point
xfs_growfs /dataKey output line: data blocks changed from A to B; B-A blocks × 4096 = bytes added. Four common mistakes:
Argument must be mount point, not device path.
Filesystem must be mounted (XFS cannot grow offline). -D grows realtime device; -d (default) grows data. Don't add -D. -n dry-run prints planned change without executing — use in scripts to detect if growth needed.
3.6 Step 5: Verification & Consistency Checks
Four-layer size match : lvs LV size, df -hT, xfs_info blocks= must reconcile. agcount from 8 → 10 proves new AGs added (concurrency 8→10). If unchanged, increment fell short of AG boundary.
Write probe : fallocate -l 8G /data/.expand_probe (metadata-only allocation, fails fast if no space).
Read-only metadata check : xfs_repair -n /dev/mapper/data-lvdata → No modify flag set.
Parameter Rationale & Coupling
① Must-change items (environment-specific)
VG name : article value data → your replacement: vgs output
LV path : article value /dev/data/lvdata → your replacement: lvs output
Multipath device : article value mpathc → your replacement: multipath -ll new entry
Mount point : article value /data → your replacement: findmnt -T <dir> AG-granularity PE count : article value 262144 → your replacement: from
xfs_info agsize × bsize / PE② Coupled items (change one → update other)
lvextendincrement ↔ xfs_growfs necessity: block device must grow first; reverse order makes xfs_growfs see no new space.
PE size ↔ all -l calculations: PE=4 MiB means +#Ext × 4 = MiB; changing PE invalidates all math. agcount ↔ write concurrency ↔ expansion rhythm: small steps on low-AG FS waste time; must add full AG each time.
Multipath path count ↔ no_path_retry: no_path_retry 30 means I/O hangs 30 retries on total path loss; must align with business timeout. /etc/fstab device path ↔ VG/LV rename: any rename requires fstab update first.
③ Performance tuning (formula-based)
Per-AG metadata overhead ≈ agsize_blks × 8 × 1.05 bytes. Baseline: 1024 GiB AG → ~2.2 GiB metadata; 10 AGs ≈ 22 GiB (0.55% of 3.95 TiB). Explains df Size vs Used+Avail gap. No need to tune imaxpct unless >200M inodes.
Stripe alignment from array: sunit = RAID stripe unit / bsize = 256 KiB / 4 KiB = 64; swidth = sunit × data disks = 64 × 2 = 128. For RAID 6 (4D+2P), data disks = 4 → swidth = 256. Parity disks excluded from swidth — common mistake.
④ Absolute Red Lines
Never lvreduce on XFS — instant corruption, no online fix.
Never mkfs.xfs -f on unrecognized LV — verify with blkid, file -s, xfs_info first.
Never pvcreate on raw /dev/sdX in multipath — creates duplicate PV labels, VG metadata chaos.
Never change pe_size — equals VG rebuild.
Never drop caches or reboot before xfs_growfs completes — mid-expansion reboot triggers log recovery; df shows old size, causing misjudgment.
Command Cheatsheet & Monitoring
4.1 One-Liners
Four-layer size match : loop over LVs, print VG/LV, LV size, dm device bytes, mount point.
Find VGs with unusable free space :
vgs --noheadings -o vg_name,vg_free_count,vg_extent_size | awk '{printf "%-10s free=%8.2f GiB PE=%4.1f MiB
", $1, $2*$3/1024, $3}'XFS fragmentation check : xfs_db -r -c 'frag -d' /dev/mapper/data-lvdata — fragmentation factor has no absolute threshold; combine with workload.
AG count delta : capture before/after agcount.
Remaining expandable space : df -B1 /data + lvs -o lv_name,lv_size,vg_free.
4.2 Prometheus Monitoring
Capacity trends (3-day forecast) + event alerts (immediate). node_exporter provides node_filesystem_*; XFS stats via node_xfs_* (enabled by default); LVM stats need lvm_exporter or textfile collector script.
Key PromQL:
# Mountpoint usage
100 - (node_filesystem_avail_bytes{fstype="xfs",mountpoint="/data"} / node_filesystem_size_bytes{fstype="xfs",mountpoint="/data"} * 100)
# 30-day linear forecast
predict_linear(node_filesystem_avail_bytes{mountpoint="/data"}[7d], 86400*30) < 0
# XFS free extent ratio
1 - (node_xfs_alloc_extents_count / (node_xfs_alloc_extents_count + node_xfs_free_extents_count))
# Inode usage
100 - (node_filesystem_files_free{fstype="xfs",mountpoint="/data"} / node_filesystem_files{fstype="xfs",mountpoint="/data"} * 100)
# Read-only remount (P1 alert)
node_filesystem_readonly{fstype="xfs",mountpoint="/data"} == 1LVM textfile script exports lvm_vg_free_bytes and lvm_pv_missing_count; lvm_pv_missing_count > 0 is P1 (VG missing PV → any LVM change fails or corrupts data).
4.3 Three Query Traps
Using node_filesystem_size_bytes as denominator underestimates usage near full (includes metadata reserve). Use (size - avail) / size plus read-only + avail dual signal. predict_linear fails on stair-step usage (batch imports). Use min_over_time(avail[30d]) as baseline. device label changes with dm minor number; always filter by mountpoint.
Three Deep-Dive Failure Cases
5.1 Duplicate PV from pvcreate on Raw Device
Symptom : vgs shows VFree 3.59T but lvextend reports
Insufficient free space: 512000 extents needed, but only 0 available.
Root cause : Admin ran pvcreate /dev/sdf (raw SCSI device) instead of /dev/mapper/mpathc. LVM added three paths ( sdf, sdg, mpathc) with same PV UUID → VG thinks it has 3×2T but physically only 1×2T. LE allocation detects conflict and refuses.
Fix : Verify same WWN via lsblk -no WWN; vgreduce data /dev/sdf /dev/sdg; pvremove /dev/sdf /dev/sdg; lvmdevices --check. If LE already allocated on wrong path, pvmove /dev/sdf first.
Script guard :
dup=$(pvs --noheadings -o pv_uuid | sort | uniq -d | wc -l); [ "${dup}" -eq 0 ] || exit 1.
5.2 xfs_growfs Given Device Path Instead of Mount Point
Symptom : Script runs xfs_growfs /dev/data/lvdata → not a mounted XFS filesystem. Script exits (due to set -e), admin retries with smaller increments, VG space exhausted, df stuck at 2.0T.
Root cause : xfs_growfs requires mount point; resize2fs accepts device path. Most admins migrate from ext4.
Fix : xfs_growfs /data. Use findmnt -n -o TARGET --source /dev/data/lvdata to auto-discover mount point in scripts.
5.3 Post-Expansion Write Performance Halved Due to RAID Mismatch
Symptom : Sequential write bandwidth drops 1.8→1.05 GB/s, random write IOPS 62k→38k. iostat shows new mpathc await 1.85ms (4× others), aqu-sz 4.6, %util 92%.
Root cause : xfs_info shows sunit=64 swidth=128 (RAID 10, 2 data disks). New LUN mpathc has optimal_io_size=134217728 (128 MiB) vs original 256 MiB — array created it as RAID 5 (4D+1P). XFS still issues 256 KiB aligned writes; on RAID 5 each write becomes read-old-data + read-parity + write-data + write-parity (4× I/O).
Fix : Short term:
pvmove /dev/mapper/mpathc; vgreduce data /dev/mapper/mpathc; pvremove /dev/mapper/mpathc(requires free space on other PVs). Long term: storage team recreates LUN as RAID 10 + 256 KiB stripe.
Prevention : Before adding to VG, compare optimal_io_size, minimum_io_size, scheduler across all mpath devices; run fio baseline — if deviation >10%, reject.
Data Migration Routes
pvmove : No downtime, limited by VG free space, used for disk swap, rebalance.
New striped LV + rsync : Yes (cutover window), no data limit, used for change striping, bsize, cross-array.
xfs_copy : Yes downtime, target ≥ source, used for fast block-level clone for verification. rsync -aHAX --numeric-ids preserves hard links ( -H critical). First sync without --delete; cutover: stop app, incremental sync with --delete, update fstab, mount new LV.
Read-Only Mount Window for xfs_repair
mount -o ro,norecovery /dev/mapper/data-lvdata /mnt/check
xfs_repair -n /dev/mapper/data-lvdata
umount /mnt/checknorecoveryskips log replay (pure read-only). For actual repair, clone via xfs_copy to new LV, repair clone, verify, then swap.
Post-Expansion fstab & Boot Verification
systemd-analyze verify /etc/fstab
systemd-fstab-generator /run/systemd/generator /etc/fstab /etc/fstab.d
ls /run/systemd/generator/*.mount | wc -l # should match fstab linesValidates syntax only; real test requires reboot drill.
Risk Matrix, Automation Script & Iron Laws
6.1 Risk Matrix
pvcreate : Scope new disk, reversible (pvremove), prerequisite wipefs clean, window any.
vgextend : Scope VG metadata, reversible (vgreduce), prerequisite no duplicate PV UUID, window any.
lvextend : Scope LV metadata, reversible (lvreduce but XFS no), prerequisite metadata backup, window low peak.
xfs_growfs : Scope filesystem, irreversible , prerequisite LV grown & mounted, window low peak.
pvmove : Scope I/O path, reversible (resume on interrupt), prerequisite target PV free space, window low peak with monitoring.
Change PE size : Scope whole VG, irreversible , prerequisite equals rebuild, window downtime.
Linear → striped : Scope whole LV, irreversible , prerequisite new LV + migrate, window downtime.
mkfs.xfs -f : Scope whole LV, irreversible , prerequisite triple-check + backup, window downtime.
Four irreversible items require formal change request.
6.2 Idempotent Expansion Script (with dry-run)
Script lvm_xfs_grow.sh does only lvextend → xfs_growfs → verify. Features: --dry-run (uses lvextend --test and xfs_growfs -n), metadata backup, pre/post AG count, write probe, duplicate PV UUID check, missing PV check, auto mount-point discovery. Dry-run output shows would be resized and identical from/to blocks; real run shows successfully resized and block delta.
6.3 Rollback Plan
Pre-lvextend : Checks fail → no rollback needed, no data risk.
lvextend space error : Insufficient free space → reduce increment or use +100%FREE, no data risk.
lvextend ok, xfs_growfs fail : df unchanged → investigate, retry xfs_growfs <mountpoint>, no data risk.
xfs_growfs busy : FS busy → re-run (reentrant), no data risk.
Write probe fail : New space unwritable → lvreduce back (safe only if FS not grown), low data risk.
Post-expand I/O perf drop : await up → pvmove new disk out, vgreduce, medium risk (needs window).
Accidental lvreduce : XFS Structure needs cleaning → only from backup , extreme data risk.
Critical: lvreduce safe only before xfs_growfs. Once FS grown, LV size bound to FS — lvreduce destroys data.
6.4 Operational Iron Laws
xfs_growfstakes mount point, not device. Use lvextend -r to auto-discover.
XFS only grows, never shrinks. Oversize → backup, rebuild, restore.
Pre-expansion must prove no duplicate PV UUIDs (multipath trap).
Watch agcount every expansion. Unchanged = concurrency unchanged = wasted effort.
New LUN must pass optimal_io_size comparison and fio baseline before joining VG. vgcfgbackup before changes, but real safety net is data backup; metadata backup only documents original state. df unchanged ≠ expansion failed (likely missed xfs_growfs); df changed ≠ success (verify agcount + write probe).
The technical depth of LVM + XFS expansion lies not in commands but in aligning four address spaces : PE granularity decides if you can allocate; AG granularity decides if allocation adds value; stripe parameters decide if it performs; multipath topology decides if prerequisites hold. Master all four layers and expansion becomes a truly copy-paste-safe operation.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
