Master Linux File Systems: From Inodes to ACLs – A Practical Production Guide

Linux File System Principles and Management: From inode to ACL Production Guide

Applicable Scenarios & Prerequisites

Applicable Scenarios : Server file system selection, performance tuning, permission management, file system failure recovery, data migration.

Prerequisites :

OS: RHEL/CentOS 7.x‑9.x, Ubuntu 18.04‑24.04

Permissions: root or sudo

Tools: e2fsprogs (ext4), xfsprogs (XFS), btrfs-progs (Btrfs), acl package

Backup: always back up critical data before file‑system operations

Knowledge: understand partitioning, mounting, inode concepts

Environment & Version Matrix

Component

RHEL/CentOS

Ubuntu/Debian

Recommended Scenario

ext4

Default support

Default file system

General use, many small files

XFS

Default (RHEL 7+)

Install xfsprogs

Large files, high concurrency, databases

Btrfs

Install btrfs-progs

Supported but not recommended for production

Snapshots, compression, small‑scale testing

ACL version

2.2.53+

2.2.53+

Fine‑grained permission control

Kernel version

3.10+ (recommended 4.18+)

4.15+ (recommended 5.4+)

-

Quick Checklist

Check file system type and mount status

Understand inode and data block allocation

Create and format file system (ext4/XFS)

Configure mount options for performance

Set up ACL for fine‑grained permissions

Run file system consistency checks and repairs

Monitor inode usage and fragmentation

Configure file system quotas

Implement snapshots and backups

Prepare migration and recovery plans

Implementation Steps

Step 1: File System Type Identification & Selection

View current file system type :

# View all mounted file system types
df -hT
# View specific device type
blkid /dev/sda1
lsblk -f
# Detailed ext4 info
tune2fs -l /dev/sda1 | grep -i "filesystem\|block size\|inode"
# XFS info
xfs_info /mount/point

Expected output :

Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda1      xfs   50G   20G   30G  40% /
/dev/sda2      ext4  100G  60G   40G  60% /data

File System Selection Decision Table :

Scenario

Recommended File System

Reason

Database (MySQL/PostgreSQL)

XFS

Excellent large‑file performance, online expansion, delayed allocation

Web server static files

ext4

Good small‑file performance, mature tooling, rich recovery utilities

Log storage

XFS

High throughput, strong concurrent write performance

Container storage (overlay2)

ext4/XFS

Both supported; XFS slightly faster

VM image storage

XFS

Strong sequential read/write for large files

Snapshot needs

Btrfs/LVM‑snapshot

Built‑in snapshot capability (use Btrfs cautiously in production)

Step 2: Understanding inode & Data Block Mechanism

inode concept :

inode stores file metadata (permissions, owner, timestamps, block pointers)

Each file consumes one inode; directories also consume inodes

Number of inodes is fixed at file system creation (adjustable for ext4)

Check inode usage :

# Show inode usage
df -i
# Show inode number of a file
ls -i file.txt
stat file.txt
# Find hard links sharing the same inode
find / -inum 123456 2>/dev/null

Expected output :

Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1      3276800 150000 3126800   5% /
/dev/sda2      6553600 500000 6053600   8% /data

inode exhaustion troubleshooting :

# Find directories with most inode consumption
for dir in /*; do echo "$dir: $(find "$dir" -xdev 2>/dev/null | wc -l) files"; done | sort -t: -k2 -nr | head -10
# Typical problem directories
find /var/spool/postfix -type f | wc -l   # mail queue
find /tmp -type f | wc -l               # temporary files
find /var/log -type f | wc -l            # log files

Solution :

# Clean small files
find /tmp -type f -mtime +7 -delete
find /var/log -name "*.log.*" -mtime +30 -delete
# If cleanup insufficient, rebuild file system with higher inode ratio
mkfs.ext4 -i 4096 /dev/sdb   # 1 inode per 4KB (default 16KB)

Step 3: Create & Format File System

ext4 creation

# Basic creation (default parameters)
mkfs.ext4 /dev/sdb1
# Production‑grade parameters
mkfs.ext4 \
    -L data-disk \
    -b 4096 \
    -i 8192 \
    -m 1 \
    -O ^has_journal \
    /dev/sdb1
# Verify parameters
tune2fs -l /dev/sdb1 | head -20

Key parameters explanation : -b 4096: block size, affects large‑file performance -i 8192: inode ratio; 4096 for many small files, 16384 for large files -m 1: reserve 1% for root (reduce on >100GB disks)

XFS creation

# Basic creation
mkfs.xfs /dev/sdb1
# Production‑grade parameters
mkfs.xfs \
    -f \
    -L data-xfs \
    -b size=4096 \
    -d agcount=32 \
    -i size=512 \
    -n size=4096 \
    /dev/sdb1
# View XFS parameters
xfs_info /dev/sdb1

XFS parameter explanation : -d agcount: number of allocation groups (usually equal to CPU cores) to improve concurrency -i size=512: inode size; 512 recommended when storing extended attributes (SELinux, ACL)

Validate file system :

# ext4 check (read‑only)
e2fsck -n /dev/sdb1
# XFS check (requires unmount)
umount /data
xfs_repair -n /dev/sdb1
mount /data

Step 4: Mount Options Optimization

Common mount options :

# View current mount options
mount | grep /data
findmnt /data
# Temporary mount for testing
mount -o noatime,nodiratime /dev/sdb1 /data
# Permanent mount (edit /etc/fstab)
cat >> /etc/fstab <<'EOF'
/dev/sdb1  /data  xfs  defaults,noatime,nodiratime,inode64,largeio  0 2
/dev/sdb2  /backup ext4 defaults,noatime,nodiratime,data=writeback,barrier=0 0 2
EOF
# Remount without downtime
mount -o remount /data
# Verify options
mount | grep /data

Mount options table :

Option

Effect

Applicable Scenario

Risk

noatime

Do not update access time

General (reduce writes)

None

nodiratime

Do not update directory access time

General

None

inode64 (XFS)

Support >2TB file systems

>2TB disks

Older kernels may not recognize

largeio (XFS)

Optimize large I/O

Databases / streaming

None

swalloc (XFS)

Delayed allocation optimization

Sequential writes

None

data=writeback (ext4)

Asynchronous writes (highest performance)

Non‑critical data with UPS

Power loss may lose data

barrier=0 (ext4)

Disable write barriers

Systems with reliable UPS/RAID cache

Power‑loss risk

discard

Enable TRIM for SSDs

SSD/NVMe

May reduce performance

SSD‑specific mount options :

# /etc/fstab example for XFS on SSD
/dev/nvme0n1p1  /data  xfs  defaults,noatime,nodiratime,discard,inode64  0 2

Verify mount options take effect :

# Touch a file and check access time
touch /data/test
stat /data/test | grep Access
# Access time should not change

Step 5: ACL Fine‑Grained Permission Control

Scenario : Multiple users/groups need different access to the same directory.

Enable ACL :

# Verify ACL support
tune2fs -l /dev/sda1 | grep "Default mount options"   # should include acl
mount | grep /data   # should include acl
# If not enabled, edit /etc/fstab and add acl option
/dev/sdb1  /data  ext4  defaults,acl  0 2
mount -o remount /data

Set ACL :

# View ACL
getfacl /data/project
# Grant user alice read/write
setfacl -m u:alice:rw /data/project/file.txt
# Grant group devops rwx
setfacl -m g:devops:rwx /data/project
# Recursive set
setfacl -R -m u:bob:rx /data/project
# Default ACL for new files
setfacl -d -m g:devops:rwx /data/project
# Remove ACL
setfacl -x u:alice /data/project/file.txt
setfacl -b /data/project/file.txt

Validate ACL :

# Example output
# user::rwx
# user:alice:rw-
# group::r-x
# group:devops:rwx
# mask::rwx
# other::r--
# Test as alice
su - alice
cat /data/project/file.txt   # should succeed

Backup & Restore ACL :

# Backup ACL
getfacl -R /data/project > /backup/project-acl.txt
# Restore ACL
setfacl --restore=/backup/project-acl.txt

Step 6: File System Check & Repair

ext4 check :

# Read‑only check (online possible, but unmount recommended)
e2fsck -n /dev/sdb1
# Automatic repair (must unmount, risky)
umount /data
umount /data
e2fsck -y /dev/sdb1   # -y answers yes automatically
mount /data
# Force full check (skip "already checked" flag)
e2fsck -f /dev/sdb1
# Verbose progress
e2fsck -fv /dev/sdb1

XFS repair :

# XFS read‑only check
xfs_repair -n /dev/sdb1
# XFS repair (must unmount)
umount /data
xfs_repair /dev/sdb1
mount /data
# If log corrupted, force clear (dangerous)
xfs_repair -L /dev/sdb1   # may lose data

Emergency read‑only mode fix :

# Detect read‑only state
dmesg | grep -i "read-only\|ext4\|xfs"
# Try remount read‑write
mount -o remount,rw /data
# If fails, boot to single‑user mode and run e2fsck -y /dev/sdb1

Preventive checks (periodic) :

# Show mount count and max mount before auto‑check
tune2fs -l /dev/sdb1 | grep -i "mount count\|maximum mount"
# Disable auto‑check (not recommended in production)
tune2fs -c 0 -i 0 /dev/sdb1
# Set custom interval (e.g., every 50 mounts)
tune2fs -c 50 /dev/sdb1

Step 7: File System Quota Management

Scenario : Limit disk usage per user or group.

Enable quota on ext4 :

# Edit /etc/fstab, add usrquota,grpquota
/dev/sdb1  /data  ext4  defaults,usrquota,grpquota  0 2
# Remount
mount -o remount /data
# Initialize quota database
quotacheck -cugm /data
# Enable quota
quotaon /data
# Verify
quotaon -p /data

Enable quota on XFS :

# Edit /etc/fstab
/dev/sdb1  /data  xfs  defaults,uquota,gquota  0 2
# Remount (quota auto‑enabled on XFS)
mount -o remount /data

Set user quota (example alice 10 GB soft, 12 GB hard):

setquota -u alice 10G 12G 0 0 /data
quota -u alice

Set group quota (example devops 50 GB soft, 60 GB hard):

setquota -g devops 50G 60G 0 0 /data
quota -g devops

View quota reports :

# All file systems
repquota -a
# Specific mount
repquota /data
# Show over‑limit users
repquota -a | grep +

XFS quota management :

# Show quota status
xfs_quota -x -c 'report -h' /data
# Set user quota
xfs_quota -x -c 'limit bsoft=10G bhard=12G alice' /data
# Set group quota
xfs_quota -x -c 'limit -g bsoft=50G bhard=60G devops' /data
# View user quota
xfs_quota -x -c 'quota -u alice' /data

Step 8: Snapshot & Backup

LVM snapshot (generic solution) :

# Create 10 GB snapshot
lvcreate -L 10G -s -n data-snapshot /dev/vg0/data
# Mount snapshot read‑only
mkdir /mnt/snapshot
mount -o ro /dev/vg0/data-snapshot /mnt/snapshot
# Backup snapshot
tar czf /backup/data-$(date +%F).tar.gz /mnt/snapshot
# Remove snapshot
umount /mnt/snapshot
lvremove /dev/vg0/data-snapshot

Btrfs snapshot (if using Btrfs) :

# Read‑only snapshot
btrfs subvolume snapshot -r /data /data/.snapshots/snap-$(date +%F)
# Read‑write snapshot
btrfs subvolume snapshot /data /data-snapshot
# List snapshots
btrfs subvolume list /data
# Delete snapshot
btrfs subvolume delete /data/.snapshots/snap-2025-10-24
# Restore from snapshot
mv /data /data.old
btrfs subvolume snapshot /data/.snapshots/snap-2025-10-24 /data

rsync incremental backup :

# Local incremental backup
rsync -avz --delete /data/ /backup/data/
# Remote backup
rsync -avz -e ssh /data/ user@backup-server:/backup/data/
# Exclude patterns
rsync -avz --exclude='*.log' --exclude='tmp/*' /data/ /backup/data/

Monitoring & Alerting

Prometheus file system monitoring

Key PromQL metrics :

# Disk usage alert (threshold 85%)
(1 - node_filesystem_avail_bytes{mountpoint=~"/|/data"} / node_filesystem_size_bytes) * 100 > 85
# inode usage alert (threshold 90%)
(1 - node_filesystem_files_free / node_filesystem_files) * 100 > 90
# Detect read‑only file systems
node_filesystem_readonly{mountpoint=~"/|/data"} == 1
# Disk read/write errors
rate(node_disk_read_errors_total[5m]) > 0 or rate(node_disk_write_errors_total[5m]) > 0

Native monitoring commands

Real‑time monitoring :

# Monitor usage every 5 seconds
watch -n 5 'df -hT && echo && df -i'
# Monitor I/O
iostat -xm 2
# Monitor open file count
watch -n 1 'cat /proc/sys/fs/file-nr'   # output: allocated  unused  max

Alert script (bash) :

#!/bin/bash
THRESHOLD=85
df -h | awk 'NR>1 {print $5,$6}' | while read usage mount; do
    usage=${usage%\%}
    if [ "$usage" -ge "$THRESHOLD" ]; then
        echo "ALERT: $mount usage $usage% >= $THRESHOLD%"
        # integrate with DingTalk/email here
    fi
done

Performance & Capacity

File system performance comparison

Test command (dd sequential write) :

# Write 1 GB file with 4 MB blocks
dd if=/dev/zero of=/data/testfile bs=4M count=256 oflag=direct
# Expected: ~200 MB/s
# Sequential read test
dd if=/data/testfile of=/dev/null bs=4M iflag=direct

Small file performance test :

# Create 10 000 small files
mkdir /data/test-small-files
cd /data/test-small-files
time for i in {1..10000}; do echo "test" > file$i.txt; done
# ext4 usually outperforms XFS for small files

Performance tuning summary table :

File System

Optimization Parameters

Applicable Scenario

ext4

data=writeback, barrier=0

Non‑critical data with UPS

ext4

noatime, nodiratime

General

ext4

dir_index (default enabled)

Large directories

XFS

inode64, largeio, swalloc

Large files, databases

XFS

logbufs=8, logbsize=256k

High‑concurrency writes

XFS

noatime, nodiratime

General

XFS advanced tuning (fstab) :

/dev/sdb1  /data  xfs  defaults,noatime,inode64,largeio,swalloc,logbufs=8,logbsize=256k  0 2

Capacity Planning

Space reservation :

OS disk (/): reserve 20%

Data disk (/data): reserve 15%

Log disk (/var/log): enable logrotate, keep 30 days

Temp disk (/tmp): regular cleanup or use tmpfs

inode capacity planning :

Small‑file workloads: 1 inode per 4 KB

Large‑file workloads: 1 inode per 32 KB

General: 1 inode per 16 KB (ext4 default)

Security & Compliance

Permission hardening :

# Set sticky bit on /tmp and /var/tmp
chmod 1777 /tmp
chmod 1777 /var/tmp
# Make critical config files immutable
chattr +i /etc/fstab
chattr +i /etc/passwd
# View attributes
lsattr /etc/fstab

Audit file access :

# Enable auditd monitoring
auditctl -w /data/sensitive -p rwa -k sensitive_access
# View audit logs
ausearch -k sensitive_access -ts recent

Encrypted file system (LUKS) :

# Create encrypted volume
cryptsetup luksFormat /dev/sdb
cryptsetup luksOpen /dev/sdb encrypted_disk
mkfs.xfs /dev/mapper/encrypted_disk
# Mount
mount /dev/mapper/encrypted_disk /data
# Unmount & close
umount /data
cryptsetup luksClose encrypted_disk

Data erasure (decommissioned disks) :

# Quick erase (not secure)
dd if=/dev/zero of=/dev/sdb bs=1M status=progress
# Secure erase (3 passes)
shred -vfz -n 3 /dev/sdb

Common Failures & Troubleshooting

Symptom

Diagnostic Command

Possible Root Cause

Quick Fix

Permanent Fix

Disk full (df 100%)

du -sh /* | sort -hr

Large files / log explosion

Delete large files

Configure log rotation

inode exhausted

df -i

Too many small files

Clean small files

Re‑create FS with more inodes

File system read‑only

dmesg | grep -i error

FS errors or disk failure

mount -o remount,rw

e2fsck -y (ext4) or xfs_repair (XFS)

Mount failure

blkid /dev/sdb1

UUID mismatch or missing device

Update /etc/fstab UUID

Use LABEL or device name

ACL not effective

mount | grep acl

ACL mount option missing

mount -o remount,acl

Update /etc/fstab

XFS log corrupted

xfs_repair -n

Power loss / hardware fault

xfs_repair -L

Replace hardware / add UPS

Deleted file still consumes space

lsof +L1

Process holds deleted file

Restart process or truncate

Graceful application shutdown

Detailed example: read‑only file system :

# 1. Confirm read‑only
mount | grep /data   # shows "ro"
# 2. Check kernel errors
dmesg | tail -50 | grep -i "xfs\|error\|i/o"
# 3. Attempt remount
mount -o remount,rw /data
# 4. If fails, unmount and repair
umount /data
xfs_repair /dev/sdb1   # or e2fsck -y /dev/sdb1
mount /data
# 5. Check hardware SMART
smartctl -H /dev/sdb
smartctl -A /dev/sdb | grep -i "reallocated\|pending"

Change & Rollback Playbooks

Pre‑change checks

#!/bin/bash
# Backup fstab
cp -p /etc/fstab /etc/fstab.bak.$(date +%F)
# Record current state
df -hT > /root/fs-check-before.txt
mount > /root/mount-check-before.txt
cat /etc/fstab > /root/fstab-check-before.txt
# Create LVM snapshot if applicable
if lvs /dev/vg0/data &>/dev/null; then
    lvcreate -L 10G -s -n data-snapshot-$(date +%F) /dev/vg0/data
fi
# Dry‑run mount test
mount -o remount,ro /data   # test unmountability
mount -o remount,rw /data
echo "Pre‑check complete. Backups stored in /root/*-before.txt and /etc/fstab.bak.*"

File system migration playbook (ext4 → XFS)

Scenario : Migrate /data from ext4 to XFS.

#!/bin/bash
set -e
SRC_DEV="/dev/sdb1"
SRC_MOUNT="/data"
DEST_DEV="/dev/sdc1"
DEST_MOUNT="/data-new"
# 1. Backup data
rsync -avxHAX --progress $SRC_MOUNT/ /backup/data-migration/
# 2. Create new FS
mkfs.xfs -f -L data-xfs $DEST_DEV
# 3. Mount new FS
mkdir -p $DEST_MOUNT
mount $DEST_DEV $DEST_MOUNT
# 4. Migrate data
rsync -avxHAX --progress $SRC_MOUNT/ $DEST_MOUNT/
# 5. Verify consistency
diff -r $SRC_MOUNT/ $DEST_MOUNT/ || echo "Data comparison complete"
# 6. Update fstab
cp /etc/fstab /etc/fstab.bak
sed -i "s|$SRC_DEV|$DEST_DEV|" /etc/fstab
sed -i "s|ext4|xfs|" /etc/fstab
# 7. Switch mount point (maintenance window)
echo "During maintenance window execute:"
echo "umount $SRC_MOUNT"
echo "mount $DEST_DEV $SRC_MOUNT"
echo "After confirming application health, delete old partition"

Rollback plan

File system corruption rollback :

# If LVM snapshot exists, merge
umount /data
lvconvert --merge /dev/vg0/data-snapshot
mount /data
# Otherwise restore from rsync backup
mkfs.xfs -f /dev/sdb1
mount /dev/sdb1 /data
rsync -avxHAX /backup/data-migration/ /data/
# Restore fstab
cp /etc/fstab.bak.$(date +%F) /etc/fstab
mount -a

Best Practices

File system selection : Use XFS for databases, ext4 for general use, Btrfs for testing or snapshot needs.

Mount options : Enable noatime,nodiratime by default; add discard for SSDs.

inode reservation : Increase inode ratio for small‑file workloads (e.g., -i 4096).

Quota management : Enable quota in multi‑user environments to prevent a single user from exhausting disk space.

Periodic checks : Run e2fsck -n or xfs_repair -n monthly for preventive health.

ACL backup : Regularly back up ACLs with getfacl -R.

Snapshot strategy : Create snapshots before changes; retain for 24 hours.

Monitoring & alerts : Use Prometheus to monitor disk usage (>85%), inode usage (>90%), and read‑only state.

Security hardening : Encrypt sensitive data with LUKS and audit critical directory access.

Migration testing : Fully rehearse file system migrations in a test environment before production.

Document version : 1.0 Test environment : RHEL 9.4, Ubuntu 24.04, Kernel 5.15+ Test date : 2025‑10