Why 'No Space Left on Device' Occurs Despite Free Disk Space: 7 Hidden Causes

When an application reports No space left on device, the first reaction is often to run df -h. If the output shows 40% free space, the issue is easily misdiagnosed as an application bug. In reality, the error comes from the kernel's ENOSPC, meaning "this allocation cannot be satisfied" — not that the entire disk's byte space is exhausted. Data blocks, inodes, user quotas, project quotas, filesystem reserved blocks, container writable layers, memory filesystems, and specific mount points can each be independently exhausted.

Secure the Failure Scene First

Do not immediately delete logs when a space alert appears. Deletion may mask inode leaks, open-but-deleted files, container writable layer bloat, or quota problems. First record the error timestamp, process, target path, and mount relationships.

date -Is
id
ps -eo pid,user,comm,args --sort=-start_time | head -n 30
journalctl --since '-15 min' -p warning..alert --no-pager

Use journalctl priority range to collect warning-to-emergency system logs. For systemd-managed applications, fetch context for the specific unit rather than only generic system logs.

systemctl status myapp.service --no-pager -l
journalctl -u myapp.service --since '-30 min' --no-pager -o short-iso
systemctl show myapp.service -p User -p Group -p RootDirectory -p RootImage

Replace myapp.service with the actual unit name. RootDirectory and RootImage reveal whether the service runs in a different root directory than the host.

If logs indicate a written file, first resolve the real filesystem hosting that path. Do not guess mount points from directory names.

target=/var/lib/myapp/data/current.db
namei -l "$target"
findmnt -T "$target" -o TARGET,SOURCE,FSTYPE,OPTIONS,SIZE,USED,AVAIL
df -hT "$target"

findmnt -T

walks the path to find the mount point carrying the target; this is more reliable than df -h without arguments. namei -l simultaneously checks ownership and permissions at each directory level. If the target file does not yet exist, change the variable to its parent directory.

df -h Only Answers the Data Block Question

A regular file involves at least two space resources: data blocks for content and inodes for metadata. df -h mainly shows data blocks; df -i shows inodes.

df -hT /var/lib/myapp
df -i /var/lib/myapp
stat -f /var/lib/myapp

If IUse% reaches 100%, creating new files returns ENOSPC even with free data blocks. Existing files can sometimes still append data because appending does not always require a new inode; operations involving temporary files, atomic renames, or log rotation fail immediately.

A non-destructive probe can distinguish "cannot create file" from "cannot extend file". The probe must run in the application's actual write directory under the application's identity.

sudo -u myapp sh -c '
  set -eu
  p=/var/lib/myapp/.enospc-probe.$$
  trap "rm -f \"$p\"" EXIT
  : >"$p"
  printf "probe
" >>"$p"
  stat "$p"
'

This creates only a tiny temporary file. If the production directory forbids probes, prepare a dedicated diagnostic directory on the same mount point rather than arbitrarily using /tmp.

Scenario 1: Inodes Exhausted by Massive Small Files

Inode exhaustion commonly stems from runaway session files, mail queues, cache shards, container layers, extraction directories, and per-request temporary files. First locate which directory level concentrates inodes. du --inodes counts inodes in a directory tree but traverses the filesystem, causing I/O pressure on huge directory counts.

sudo du -x --inodes --max-depth=2 /var 2>/dev/null |
  sort -nr | head -n 30

-x

prevents crossing other filesystems, avoiding inclusion of network mounts or child mount points. Start with a shallow --max-depth to find the general direction, then narrow down on suspicious directories.

When a directory may contain millions of files, do not run ls -l which sorts entirely first. Use find for streaming counts, limited to the current filesystem.

sudo find /var/lib/myapp -xdev -type f -printf '%h
' 2>/dev/null |
  sort | uniq -c | sort -nr | head -n 30

This still traverses the tree, suitable for off-peak execution. If the hotspot directory is already known, count only direct children to reduce scan scope.

hot=/var/lib/myapp/cache
find "$hot" -mindepth 1 -maxdepth 1 -printf '.' 2>/dev/null | wc -c
find "$hot" -mindepth 1 -maxdepth 1 -type f \
  -printf '%TY-%Tm-%Td %TH:%TM %p
' 2>/dev/null | head

Before cleanup, verify file lifecycles and sample-check whether processes still use them. Do not judge deletability solely by "old modification time".

candidate=/var/lib/myapp/cache/example.tmp
stat "$candidate"
sudo lsof -- "$candidate"
sudo fuser -v "$candidate"

If application documentation explicitly states cache is rebuildable, first generate an inventory and assess count and capacity impact. The following only generates candidates, does not delete.

cutoff_days=14
candidate_list=/var/tmp/myapp-cache-candidates.txt
sudo find /var/lib/myapp/cache -xdev -type f \
  -mtime +"$cutoff_days" -print0 \
  | sudo tee /var/tmp/myapp-cache-candidates.nul >/dev/null
tr '\0' '
' </var/tmp/myapp-cache-candidates.nul > "$candidate_list"
wc -l "$candidate_list"
sed -n '1,20p' "$candidate_list"

Actual cleanup is data deletion. Obtain business confirmation or rebuild proof before execution, retain the candidate list, canary one time slice first, and observe application error rate and inodes. Filenames containing newlines break text lists, so real execution uses the NUL-delimited raw list.

# High risk: only after candidates audited and cache confirmed rebuildable
sudo xargs -0 -r -n 200 rm -- \
  </var/tmp/myapp-cache-candidates.nul

df -i /var/lib/myapp/cache
sudo -u myapp test -w /var/lib/myapp/cache

Deleting regular files cannot be directly "rolled back". When rollback capability is needed, move a small batch of candidates to an isolation directory on the same filesystem; cross-filesystem moves become copy-then-delete, which is slow and may consume space again.

sudo install -d -m 0700 /var/lib/myapp/.quarantine
sudo mv -- /var/lib/myapp/cache/example.tmp \
  /var/lib/myapp/.quarantine/

# To rollback after business verification:
sudo mv -- /var/lib/myapp/.quarantine/example.tmp \
  /var/lib/myapp/cache/

Scenario 2: Application's Mount Point Full, But You're Looking at Another Disk

/

having 40% free does not mean /var, /var/lib/docker, /run, or a bind mount has space. A directory may also be covered by a new mount, hiding data written before the mount.

findmnt -R /var
lsblk -o NAME,FSTYPE,SIZE,FSAVAIL,FSUSE%,MOUNTPOINTS
df -hT -x tmpfs -x devtmpfs

For the application process, also check its own mount namespace. Containers or services with PrivateMounts=yes may see different mount relationships than the current shell.

pid=$(systemctl show -p MainPID --value myapp.service)
sudo nsenter -t "$pid" -m -- findmnt -T /var/lib/myapp
sudo nsenter -t "$pid" -m -- df -hT /var/lib/myapp

MainPID

must be greater than 0; do not pass 0 to nsenter when the service is not running. If the process is in a container, use paths as seen inside the process namespace.

To check whether a directory is covered by a mount, compare device and inode before and after the mount. Do not unmount in production just to "take a look" — this interrupts processes accessing it. A read-only bind mount can view the host root filesystem but still cannot penetrate an already-covered directory; the safest approach is to inspect the underlying filesystem in a rescue environment or maintenance window.

findmnt /var/lib/myapp -o TARGET,SOURCE,ROOT,FSTYPE,OPTIONS
mountpoint -q /var/lib/myapp && echo '独立挂载点'
stat -c 'device=%D inode=%i path=%n' /var/lib/myapp

Scenario 3: ext4 Reserved Blocks Cause Ordinary Users to See ENOSPC Early

ext2/3/4 typically reserve a portion of data blocks for privileged users so the system can still write logs, allow logins, and repair when space is tight. df 's available space already accounts for reserved blocks, but actual availability differs by identity. First confirm filesystem type and reserved block settings.

src=$(findmnt -no SOURCE -T /var/lib/myapp)
fstype=$(findmnt -no FSTYPE -T /var/lib/myapp)
printf 'source=%s fstype=%s
' "$src" "$fstype"
sudo tune2fs -l "$src" | grep -E \
  'Block count|Reserved block count|Reserved block percentage|Block size'

tune2fs

applies only to ext series filesystems; in LVM or dm-crypt environments SOURCE may be a device-mapper path. Do not run modification commands on XFS, Btrfs, or unknown devices.

Lowering the reserved percentage increases ordinary-user available space but reduces the failure buffer. Root filesystems should not be casually set to 0; large pure-data volumes can be lowered after evaluation. Record original values and device UUID before modifying.

# High-risk config change: confirm target is ext4 data volume, record rollback value
sudo blkid "$src"
sudo tune2fs -l "$src" | grep -E 'Reserved block count|Reserved block percentage'

# Example: adjust reserved percentage to 1%, do not copy directly to root filesystem
sudo tune2fs -m 1 "$src"
df -hT /var/lib/myapp

Rollback is simply re-running tune2fs -m original_value device. Adjusting reserved blocks does not solve inode, quota, or child mount point exhaustion.

Scenario 4: Quota Limits the User, Group, or Project

The filesystem overall has space, but the application user may exceed quota. First confirm from mount options whether user, group, or project quota is enabled.

findmnt -T /var/lib/myapp -o TARGET,SOURCE,FSTYPE,OPTIONS
sudo quota -vs myapp 2>/dev/null || true
sudo repquota -a 2>/dev/null | sed -n '1,80p'

ext4 commonly uses usrquota, grpquota; XFS commonly uses uquota, gquota, prjquota. Soft limits allow brief overrun with a grace period; hard limits immediately reject allocation.

XFS project quotas often limit directories rather than Unix users. The following commands read-only view project definitions and usage.

sudo xfs_quota -x -c 'state' /var/lib/myapp
sudo xfs_quota -x -c 'report -p -h' /var/lib/myapp
sudo xfs_quota -x -c 'quota -p -h myapp' /var/lib/myapp

Adjusting quota changes resource isolation boundaries; must first confirm capacity budget, neighbor tenant impact, and original limits. Below shows operation form; do not use example values directly in production.

# High risk: record original values, validate in test project first; example sets soft/hard to 80G/90G
sudo xfs_quota -x -c 'limit -p bsoft=80g bhard=90g myapp' /var/lib/myapp
sudo xfs_quota -x -c 'quota -p -h myapp' /var/lib/myapp

# Rollback: rewrite original bsoft/bhard values recorded at query time

If short-term limit increase is only firefighting, establish an expiration reclamation plan simultaneously, otherwise temporary allowance becomes permanent.

Scenario 5: File Deleted but Space Still Held by Process

Unix deletion removes the directory entry. While a process holds the file descriptor, data blocks are released only when the last reference closes. At this point du cannot see the file, but df shows space occupied; however this scenario usually makes df genuinely near full and cannot explain an arbitrary "40% available" filesystem unless the wrong mount point was examined.

sudo lsof +L1 2>/dev/null \
  | awk 'NR==1 || $7 ~ /^[0-9]+$/ {print}' \
  | sort -k7,7nr | head -n 30

+L1

filters open files with link count less than 1. After confirming PID, service name, and file size, prefer letting the application reopen logs normally — e.g., send the application's supported reopen signal or execute log rotation. Do not send HUP to arbitrary processes without confirmation.

sudo logrotate -d /etc/logrotate.conf
sudo logrotate -f /etc/logrotate.d/myapp
sleep 2
sudo lsof +L1 | grep myapp || true
df -hT /var/lib/myapp

logrotate -d

is debug mode, does not rotate; before forced rotation confirm the config's postrotate can make the program reopen files. If the application does not support reopen, schedule a rolling restart instead of directly truncating /proc/PID/fd/N. The latter modifies a file still in use, potentially breaking audit or business data.

Scenario 6: tmpfs, Shared Memory, or Container Writable Layer Exhausted

/run

, /dev/shm, and some /tmp are tmpfs, with capacity from memory/swap limits, unrelated to physical disk remaining.

findmnt -t tmpfs -o TARGET,SOURCE,SIZE,USED,AVAIL,OPTIONS
df -hT /run /dev/shm /tmp 2>/dev/null
du -x -h --max-depth=1 /dev/shm 2>/dev/null | sort -h

The / seen inside a container is often an overlay writable layer. Host data disk having space does not mean Docker data-root, containerd snapshotter, or Pod ephemeral-storage has space.

docker info --format 'DockerRootDir={{.DockerRootDir}} Driver={{.Driver}}'
docker system df -v
findmnt -T "$(docker info --format '{{.DockerRootDir}}')"

docker system prune

deletes unused objects and is destructive; should not be used as a diagnostic command. First list containers, images, volumes, and build cache, confirm rollback-required images are still pullable from trusted registries, then clean by object type.

In Kubernetes, simultaneously check node filesystem and Pod ephemeral storage limits.

kubectl get pod -n myns mypod -o jsonpath='{range .spec.containers[*]}{.name}{" request="}{.resources.requests.ephemeral-storage}{" limit="}{.resources.limits.ephemeral-storage}{"
"}{end}'
kubectl describe node worker-01 | sed -n '/Allocated resources:/,/Events:/p'
kubectl get events -A --field-selector reason=Evicted --sort-by=.lastTimestamp

Do not verify storage issues by deleting Pods unless controller, replica count, PodDisruptionBudget, and data persistence are all confirmed. Deletion also destroys writable layer evidence.

Scenario 7: Preallocation, Metadata, and Filesystem State

When an application calls fallocate to preallocate a large file, it requests many data blocks at once; current space fitting small files does not guarantee satisfying this allocation. Sparse file logical size also does not represent actual usage.

file=/var/lib/myapp/data/example.db
stat -c 'size=%s bytes blocks=%b block_size=%B' "$file"
du -h "$file"
du -h --apparent-size "$file"

size

is logical length; blocks × 512 approximates actual allocation. Databases, VM images, and log files must be judged with application behavior; cannot be directly truncated.

Kernel logs may indicate XFS shutdown, ext4 errors, I/O errors, or read-only remount. These are sometimes wrapped by applications as generic write failures.

journalctl -k --since '-1 hour' --no-pager \
  | grep -Ei 'enospc|no space|ext4|xfs|btrfs|i/o error|read-only|quota'
findmnt -T /var/lib/myapp -o TARGET,SOURCE,FSTYPE,OPTIONS

If filesystem errors or read-only remount appear, stop write propagation and formulate a maintenance plan per filesystem type. fsck cannot be arbitrarily run on mounted read-write filesystems; XFS does not use traditional fsck for repair. First take storage snapshots or backups, schedule unmount window, then use the corresponding official tool's check mode.

# Check-only command examples, must run on offline ext4 device in maintenance window
sudo e2fsck -fn /dev/mapper/vg-data

# XFS read-only check, also requires filesystem unmounted
sudo xfs_repair -n /dev/mapper/vg-xfsdata

-n

/ -f -n are for read-only check; does not mean they can run on arbitrary production devices. Before actual repair, preserve diagnostic output, verify backup restorability, and prepare for extended downtime.

Build a Judgment Chain That Doesn't Go Astray

Why df , du , and Application Perspective Often Disagree

df

reads allocated and available blocks from the filesystem superblock, answering the whole-filesystem perspective; du traverses visible directory entries and sums file usage, answering the current namespace and permission perspective. Their discrepancy does not automatically indicate filesystem corruption. Deleted-but-open files count only in df; files hidden by other mounts are invisible in the current directory tree; sparse file logical size differs from actual block count; snapshots and CoW shared blocks cannot be accurately expressed by simple directory summation.

For XFS, Btrfs, etc., metadata also consumes space. A filesystem may appear to have data space, but specific allocation groups, metadata block groups, or devices may face local pressure. Do not substitute "total remaining ratio" for filesystem-specific status. XFS can first view overall and allocation group info; Btrfs must distinguish device usage from filesystem usage:

fstype=$(findmnt -no FSTYPE -T /var/lib/myapp)
case "$fstype" in
  xfs)
    xfs_info /var/lib/myapp
    sudo xfs_db -r -c 'freesp -s' "$(findmnt -no SOURCE -T /var/lib/myapp)"
    ;;
  btrfs)
    sudo btrfs filesystem usage -T /var/lib/myapp
    sudo btrfs device usage /var/lib/myapp
    ;;
esac

xfs_db -r

uses read-only mode but still confirm the argument is an XFS device. Btrfs Data, Metadata, and System block groups allocate separately; device not full does not mean metadata has free chunks. Btrfs qgroup may also let a subvolume hit referenced space limit, manifesting as overall space available but local write failure.

sudo btrfs quota status /var/lib/myapp
sudo btrfs qgroup show -reF /var/lib/myapp
sudo btrfs subvolume show /var/lib/myapp

Do not arbitrarily run btrfs balance at the incident scene. Balance rewrites large amounts of blocks; when space is critically tight or devices unhealthy, it may increase I/O pressure. First read usage layout, confirm backups and device health, then target data or metadata with restricted filters, and validate cancel/recovery procedures in a test environment.

Inode Problems Must Be Solved at the Generation Source

ext4 inode count is usually fixed at filesystem creation and cannot be increased online like regular capacity. Long-term solution is not daily batch deletion but changing small-file generation patterns or rebuilding the filesystem: move massive sessions into a TTL-enabled database or cache, merge small objects into combined storage, limit task output count, set lifecycle cleanup, and alert on creation rate. If ext4 rebuild is mandatory, choose appropriate bytes-per-inode on the new volume based on average file size, complete data verification and switchover before reclaiming the old volume.

Migration is high-risk. At minimum requires capacity and inode dual verification, application write-stop or incremental sync mechanism, permission/ACL/xattr preservation, checksum comparison, mount parameter recording, and a fast rollback plan to the old volume. Merely using cp -r often loses hard links, ACLs, extended attributes, or sparse characteristics. Proper tools include validated-parameter rsync, storage snapshot replication, or application-native backup/restore, but choice depends on data semantics.

Logs and temporary files should be handed to explicit lifecycle management, not rely on on-call manual deletion. When checking journald usage, first view statistics, then decide vacuum per organizational retention:

journalctl --disk-usage
systemd-analyze cat-config systemd/journald.conf \
  | grep -E '^(SystemMaxUse|SystemKeepFree|RuntimeMaxUse|MaxRetentionSec)='

# Only estimate archived logs releasable by time-based cleanup, do not delete
journalctl --vacuum-time=14d --dry-run 2>/dev/null || true

Not all systemd versions support vacuum dry-run; unsupported versions fail, the || true here is only for compatibility check. Real vacuum irreversibly deletes archived logs; must first satisfy audit retention, confirm logs are centrally collected, and record deletion scope. Application logs should check logrotate frequency, rotate, maxsize, compression, and reopen behavior to avoid "rotation succeeded but process continues writing old file".

Why Atomic Writes Need More Space Than They Appear

Many applications update config or database fragments not by in-place overwrite but by writing a temporary file in the same directory, executing fsync, then rename over the old file. This guarantees power-loss consistency yet means briefly needing space for both old and new versions plus an extra inode. Compression, backup, database compaction, and container image unpacking have similar peaks. Capacity planning cannot rely only on steady-state file sizes; must account for peak usage during releases, backups, restores, and compaction.

Directories themselves grow. Millions of directory entries increase lookup and metadata update cost even if file contents are small. Hashing files into multi-level directories improves single-directory operations but does not reduce total inode count; if application deletion speed lags creation speed, sharding only postpones failure.

Capacity Alerts Should Cover "How Long Until Exhaustion"

Fixed thresholds only detect already near-exhaustion states. More practical monitoring includes current level, growth rate, and predicted time-to-exhaustion. Compute separately for data blocks and inodes, using sufficiently long observation windows during periodic jobs like backups and batch processing. Growth is non-linear; prediction should only provide lead time, not replace hard thresholds.

predict_linear(
  node_filesystem_avail_bytes{mountpoint="/var/lib/myapp"}[6h],
  24 * 3600
) < 0

predict_linear(
  node_filesystem_files_free{mountpoint="/var/lib/myapp"}[6h],
  24 * 3600
) < 0

Prediction window should cover typical workload cycles; systems with daily cycles may false-alert with 6-hour linear prediction. Alert annotations must include recent growth rate, target mount point, filesystem type, largest directory, and responsible service so on-call personnel don't have to rediscover context.

Core checks can be converged into a single read-only script. It does not perform auto-remediation because handling risks differ greatly across root causes.

#!/usr/bin/env bash
set -Eeuo pipefail

target=${1:-}
if [[ -z "$target" || ! -e "$target" ]]; then
  echo "Usage: $0 <existing file or directory>" >&2
  exit 2
fi

echo '== identity =='
date -Is
id

echo '== mount =='
findmnt -T "$target" -o TARGET,SOURCE,FSTYPE,OPTIONS,SIZE,USED,AVAIL

echo '== blocks =='
df -hT "$target"

echo '== inodes =='
df -i "$target"

echo '== path =='
namei -l "$target"

echo '== deleted-open files on same host =='
lsof +L1 2>/dev/null | head -n 20 || true

echo '== recent kernel storage messages =='
journalctl -k --since '-30 min' --no-pager 2>/dev/null \
  | grep -Ei 'enospc|no space|quota|ext4|xfs|btrfs|i/o error|read-only' \
  | tail -n 50 || true

Save as enospc-check.sh and run with the actual target directory. lsof may have overhead on busy hosts; if strict control needed, remove that segment and run separately during maintenance.

Post-Fix Verification Must Go Beyond df

After root cause remediation, verify create, write, sync, and delete as the application identity, while confirming error rates, logs, and resource levels recover. sync system call necessity depends on the application; below uses Python to fsync a single probe file without triggering a full-system sync.

sudo -u myapp python3 - <<'PY'
import os
from pathlib import Path

p = Path('/var/lib/myapp/.write-probe')
try:
    with p.open('wb') as f:
        f.write(b'enospc verification
')
        f.flush()
        os.fsync(f.fileno())
    print(p.stat())
finally:
    p.unlink(missing_ok=True)
PY

Then re-check blocks, inodes, quotas, and service logs, and observe at least one normal business cycle.

df -hT /var/lib/myapp
df -i /var/lib/myapp
sudo quota -vs myapp 2>/dev/null || true
journalctl -u myapp.service --since '-10 min' --no-pager \
  | grep -Ei 'enospc|no space|quota|write|error' || true

Monitoring should at minimum cover both data blocks and inodes. With node_exporter, compute available bytes ratio and inode free ratio separately, excluding read-only and pseudo filesystems.

100 * node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
  / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}

100 * node_filesystem_files_free{fstype!~"tmpfs|overlay|squashfs"}
  / node_filesystem_files{fstype!~"tmpfs|overlay|squashfs"}

Alerts must correlate by instance, device, mountpoint, and fstype, not just hostname. Also combine quota metrics, container ephemeral-storage, write error rates, and capacity growth velocity to avoid again falling for the "disk clearly has 40%" illusion.

Final Judgment Reference

Symptom: df -h has space, df -i 100% — Primary Evidence: Inode usage %, hotspot directory file count — Common Remediation: Cleanup small files by lifecycle, fix generation strategy

Symptom: Root partition has space, target still ENOSPC — Primary Evidence: findmnt -T, process mount namespace — Common Remediation: Handle the real target mount point

Symptom: Root writable, application user not writable — Primary Evidence: Quota, ext4 reserved blocks, application identity probe — Common Remediation: Adjust quota or release that user's usage

Symptom: du small, df large — Primary Evidence: lsof +L1 — Common Remediation: Let process close or reopen files normally

Symptom: Host disk has space, container fails — Primary Evidence: Overlay, data-root, ephemeral-storage — Common Remediation: Clean or expand correct container storage layer

Symptom: Kernel shows filesystem errors — Primary Evidence: Kernel journal, mount options — Common Remediation: Stop writes, snapshot, offline check and repair

The correct question for ENOSPC is not "which disk is full" but "in what namespace, on what filesystem, with what identity, requesting which limited resource did this write fail?" Once these four dimensions are confirmed, the 40% remaining space is no longer a contradiction but merely an unrelated or incomplete metric.