Why 'No Space Left on Device' When Disk Shows 40% Free?

When an application reports No space left on device, the first reaction is often to run df -h. If the output shows 40% free space, the issue is easily misdiagnosed as an application bug. In reality, this error comes from the kernel's ENOSPC return code, meaning "this allocation cannot be satisfied" — not that the entire disk's byte space is exhausted. Data blocks, inodes, user quotas, project quotas, filesystem reserved blocks, container writable layers, memory filesystems, and specific mount points can each be independently exhausted.

The key to troubleshooting is not proving "disk has space" but confirming where the failing write occurs, what resource it needs to allocate, under which identity, and why the kernel refuses that allocation.

First, Freeze the Failure Scene

Do not delete logs immediately upon seeing a space alert. Deletion may hide inode leaks, open-but-deleted files, container writable layer bloat, or quota issues. First, record the error time, process, target path, and mount relationships.

date -Is
id
ps -eo pid,user,comm,args --sort=-start_time | head -n 30
journalctl --since '-15 min' -p warning..alert --no-pager

The journalctl priority range collects system logs from warning to alert. If the application is managed by systemd, fetch context for the specific unit instead of only generic system logs.

systemctl status myapp.service --no-pager -l
journalctl -u myapp.service --since '-30 min' --no-pager -o short-iso
systemctl show myapp.service -p User -p Group -p RootDirectory -p RootImage

Replace myapp.service with the actual unit name. RootDirectory and RootImage reveal whether the service runs in a different root directory than the host.

If logs indicate a written file, first resolve the real filesystem holding that path. Do not guess mount points from directory names.

target=/var/lib/myapp/data/current.db
namei -l "$target"
findmnt -T "$target" -o TARGET,SOURCE,FSTYPE,OPTIONS,SIZE,USED,AVAIL
df -hT "$target"

findmnt -T

walks the path to find the mount point carrying the target; this is more reliable than a plain df -h. namei -l also checks ownership and permissions at each directory level. If the target file does not yet exist, change the variable to its parent directory.

df -h Only Answers the Data Block Question

A regular file involves at least two space resources: data blocks storing content and inodes storing metadata. df -h mainly shows data blocks; df -i shows inodes.

df -hT /var/lib/myapp
df -i /var/lib/myapp
stat -f /var/lib/myapp

If IUse% reaches 100%, creating new files returns ENOSPC even if data blocks are free. Existing files can sometimes still append data because appending does not always require a new inode; operations involving temporary files, atomic renames, or log rotation fail immediately.

A non-destructive probe can distinguish "cannot create file" from "cannot extend file". The probe must run in the application's actual write directory and under the application's identity.

sudo -u myapp sh -c '
  set -eu
  p=/var/lib/myapp/.enospc-probe.$$
  trap "rm -f \"$p\"" EXIT
  : >"$p"
  printf "probe
" >>"$p"
  stat "$p"
'

This creates only a tiny temporary file. If the production directory forbids any probe, prepare a dedicated diagnostic directory on the same mount point rather than arbitrarily using /tmp.

Scenario 1: Inodes Exhausted by Massive Small Files

Inode exhaustion commonly stems from runaway session files, mail queues, cache shards, container layers, extraction directories, and per-request temporary files. First, locate which directory level concentrates the inodes. du --inodes counts inodes in a directory tree but traverses the filesystem, causing I/O pressure when directory counts are huge.

sudo du -x --inodes --max-depth=2 /var 2>/dev/null |
  sort -nr | head -n 30

-x

prevents crossing other filesystems, avoiding inclusion of network mounts or child mount points. Start with a shallow --max-depth to find the general direction, then narrow down suspicious directories layer by layer.

When a directory may contain millions of files, do not run ls -l which sorts the entire list first. Use find for streaming counting, limited to the current filesystem.

sudo find /var/lib/myapp -xdev -type f -printf '%h
' 2>/dev/null |
  sort | uniq -c | sort -nr | head -n 30

This still traverses the tree, suitable for off-peak execution. If the hotspot directory is already known, count only its immediate children to reduce scan scope.

hot=/var/lib/myapp/cache
find "$hot" -mindepth 1 -maxdepth 1 -printf '.' 2>/dev/null | wc -c
find "$hot" -mindepth 1 -maxdepth 1 -type f \
  -printf '%TY-%Tm-%Td %TH:%TM %p
' 2>/dev/null | head

Before cleanup, confirm file lifecycles and sample-check whether processes still hold them. Do not judge deletability solely by "older modification time".

candidate=/var/lib/myapp/cache/example.tmp
stat "$candidate"
sudo lsof -- "$candidate"
sudo fuser -v "$candidate"

If application documentation explicitly states cache is rebuildable, generate a candidate list and assess count and capacity first. The following only generates candidates, does not delete.

cutoff_days=14
candidate_list=/var/tmp/myapp-cache-candidates.txt
sudo find /var/lib/myapp/cache -xdev -type f \
  -mtime +"$cutoff_days" -print0 \
  | sudo tee /var/tmp/myapp-cache-candidates.nul >/dev/null
tr '\0' '
' </var/tmp/myapp-cache-candidates.nul >"$candidate_list"
wc -l "$candidate_list"
sed -n '1,20p' "$candidate_list"

Actual cleanup is data deletion. Before execution, obtain business confirmation or rebuild proof, retain the candidate list, canary one time slice first, and observe application error rate and inodes. Filenames containing newlines make text lists incomplete, so real execution uses the NUL-delimited raw list.

# High risk: execute only after candidates audited and cache confirmed rebuildable
sudo xargs -0 -r -n 200 rm -- \
  </var/tmp/myapp-cache-candidates.nul

df -i /var/lib/myapp/cache
sudo -u myapp test -w /var/lib/myapp/cache

Deleting regular files cannot be directly "rolled back". When rollback is needed, move a small batch of candidates to an isolation directory on the same filesystem; cross-filesystem moves become copy-then-delete, which is slow and may consume space again.

sudo install -d -m 0700 /var/lib/myapp/.quarantine
sudo mv -- /var/lib/myapp/cache/example.tmp \
  /var/lib/myapp/.quarantine/

# Rollback after business verification:
sudo mv -- /var/lib/myapp/.quarantine/example.tmp \
  /var/lib/myapp/cache/

Scenario 2: Application's Mount Point Full, But You're Looking at Another Disk

/

having 40% free does not mean /var, /var/lib/docker, /run, or a bind mount have space. A directory may also be covered by a new mount, hiding data written before the mount.

findmnt -R /var
lsblk -o NAME,FSTYPE,SIZE,FSAVAIL,FSUSE%,MOUNTPOINTS
df -hT -x tmpfs -x devtmpfs

For the application process, also check its own mount namespace. Containers or services with PrivateMounts=yes may see different mount relationships than the current shell.

pid=$(systemctl show -p MainPID --value myapp.service)
sudo nsenter -t "$pid" -m -- findmnt -T /var/lib/myapp
sudo nsenter -t "$pid" -m -- df -hT /var/lib/myapp

MainPID

must be greater than 0; do not pass 0 to nsenter when the service is not running. If the process is in a container, use paths as seen inside the process namespace.

To check whether a directory is covered by a mount, compare device and inode before and after the mount. Do not unmount a production mount point just to "take a look" — this interrupts processes accessing it. A read-only bind mount can view the host root filesystem, but still cannot see through an already-covered directory; the safest is to inspect the underlying filesystem in a rescue environment or maintenance window.

findmnt /var/lib/myapp -o TARGET,SOURCE,ROOT,FSTYPE,OPTIONS
mountpoint -q /var/lib/myapp && echo 'independent mount point'
stat -c 'device=%D inode=%i path=%n' /var/lib/myapp

Scenario 3: ext4 Reserved Blocks Make Ordinary Users See ENOSPC Early

ext2/3/4 typically reserve a portion of data blocks for privileged users so the system can still write logs, allow logins, and repair when space is tight. df 's available space already accounts for reserved blocks, but actual availability differs by identity. First confirm filesystem type and reserved block settings.

src=$(findmnt -no SOURCE -T /var/lib/myapp)
fstype=$(findmnt -no FSTYPE -T /var/lib/myapp)
printf 'source=%s fstype=%s
' "$src" "$fstype"
sudo tune2fs -l "$src" | grep -E \
'Block count|Reserved block count|Reserved block percentage|Block size'

tune2fs

applies only to ext series filesystems; in LVM or dm-crypt environments SOURCE may be a device mapper path. Do not run modification commands on XFS, Btrfs, or unknown devices.

Lowering the reserved percentage increases space for ordinary users but reduces the failure buffer. Root filesystems should not be casually set to 0; large pure data volumes can be lowered after evaluation. Record original values and device UUID before modifying.

# High-risk config change: confirm target is ext4 data volume and record rollback values
sudo blkid "$src"
sudo tune2fs -l "$src" | grep -E 'Reserved block count|Reserved block percentage'

# Example: adjust reserved percentage to 1%, do not copy directly to root filesystem
sudo tune2fs -m 1 "$src"
df -hT /var/lib/myapp

Rollback is simply running tune2fs -m original_value device again. Adjusting reserved blocks does not solve inode, quota, or child mount point exhaustion.

Scenario 4: Quotas Limit User, Group, or Project

The filesystem overall has space, but the application user may exceed quota. First, check mount options to see if user, group, or project quotas are enabled.

findmnt -T /var/lib/myapp -o TARGET,SOURCE,FSTYPE,OPTIONS
sudo quota -vs myapp 2>/dev/null || true
sudo repquota -a 2>/dev/null | sed -n '1,80p'

ext4 commonly uses usrquota, grpquota; XFS commonly uses uquota, gquota, prjquota. Soft limits allow brief overrun with a grace period; hard limits reject allocation immediately.

XFS project quotas often limit directories rather than Unix users. The following commands read-only view project definitions and usage.

sudo xfs_quota -x -c 'state' /var/lib/myapp
sudo xfs_quota -x -c 'report -p -h' /var/lib/myapp
sudo xfs_quota -x -c 'quota -p -h myapp' /var/lib/myapp

Adjusting quotas changes resource isolation boundaries; must first confirm capacity budget, neighbor tenant impact, and original limits. Below shows operation form; do not use example values directly in production.

# High risk: record original values, validate in test project first; example sets soft/hard to 80G/90G
sudo xfs_quota -x -c 'limit -p bsoft=80g bhard=90g myapp' /var/lib/myapp
sudo xfs_quota -x -c 'quota -p -h myapp' /var/lib/myapp

# Rollback: write back original bsoft/bhard values recorded at query time

If short-term limit increase is only for firefighting, establish an expiration reclamation plan simultaneously, otherwise temporary allowances become permanent.

Scenario 5: File Deleted but Space Still Held by Process

Unix deletes the directory entry. While a process holds the file descriptor, data blocks are freed only when the last reference closes. At this point du cannot see the file, but df shows space used; however this scenario usually makes df genuinely near full and cannot explain an arbitrary "40% available" filesystem unless you are looking at the wrong mount point.

sudo lsof +L1 2>/dev/null \
  | awk 'NR==1 || $7 ~ /^[0-9]+$/ {print}' \
  | sort -k7,7nr | head -n 30

+L1

filters open files with link count less than 1. After confirming PID, service name, and file size, prefer letting the application reopen logs normally — e.g., send the application's supported reopen signal or execute log rotation. Do not send HUP to arbitrary processes without confirmation.

sudo logrotate -d /etc/logrotate.conf
sudo logrotate -f /etc/logrotate.d/myapp
sleep 2
sudo lsof +L1 | grep myapp || true
df -hT /var/lib/myapp

logrotate -d

is debug mode, does not rotate; before forced rotation confirm the config's postrotate can make the program reopen the file. If the application does not support reopen, schedule a rolling restart instead of directly truncating /proc/PID/fd/N. The latter modifies a file still in use, potentially breaking audit or business data.

Scenario 6: tmpfs, Shared Memory, or Container Writable Layer Exhausted

/run

, /dev/shm, and some /tmp are tmpfs, capacity sourced from memory/swap limits, unrelated to physical disk remaining.

findmnt -t tmpfs -o TARGET,SOURCE,SIZE,USED,AVAIL,OPTIONS
df -hT /run /dev/shm /tmp 2>/dev/null
du -x -h --max-depth=1 /dev/shm 2>/dev/null | sort -h

The / seen inside a container is often an overlay writable layer. Host data disk having space does not mean Docker data-root, containerd snapshotter, or Pod ephemeral-storage have space.

docker info --format 'DockerRootDir={{.DockerRootDir}} Driver={{.Driver}}'
docker system df -v
findmnt -T "$(docker info --format '{{.DockerRootDir}}')"

docker system prune

deletes unused objects and is destructive; should not be used as a diagnostic command. First list containers, images, volumes, and build caches, confirm required images are still pullable from trusted registries, then clean by object type.

In Kubernetes, simultaneously check node filesystem and Pod ephemeral storage limits.

kubectl get pod -n myns mypod -o jsonpath='{range .spec.containers[*]}{.name}{" request="}{.resources.requests.ephemeral-storage}{" limit="}{.resources.limits.ephemeral-storage}{"
"}{end}'
kubectl describe node worker-01 | sed -n '/Allocated resources:/,/Events:/p'
kubectl get events -A --field-selector reason=Evicted --sort-by=.lastTimestamp

Do not verify storage issues by deleting Pods unless controller, replica count, PodDisruptionBudget, and data persistence are all confirmed. Deletion also destroys writable layer evidence.

Scenario 7: Preallocation, Metadata, and Filesystem State

When an application calls fallocate to preallocate a large file, it requests many data blocks at once; current space may fit small files but not this allocation. Sparse file logical size also does not reflect actual usage.

file=/var/lib/myapp/data/example.db
stat -c 'size=%s bytes blocks=%b block_size=%B' "$file"
du -h "$file"
du -h --apparent-size "$file"

size

is logical length; blocks × 512 approximates actual allocated bytes. Databases, VM images, and log files must be judged in context of application behavior, not simply truncated.

Kernel logs may indicate XFS shutdown, ext4 errors, I/O errors, or read-only remounts. These issues are sometimes wrapped by applications into generic write failures.

journalctl -k --since '-1 hour' --no-pager \
  | grep -Ei 'enospc|no space|ext4|xfs|btrfs|i/o error|read-only|quota'
findmnt -T /var/lib/myapp -o TARGET,SOURCE,FSTYPE,OPTIONS

If filesystem errors or read-only remounts appear, stop write propagation and formulate a maintenance plan per filesystem type. fsck cannot be run arbitrarily on mounted read-write filesystems; XFS does not use traditional fsck for repair. First take storage snapshots or backups, schedule an unmount window, then use the corresponding official tool's check mode.

# Check-only command examples, must run on offline ext4 device in maintenance window
sudo e2fsck -fn /dev/mapper/vg-data

# XFS read-only check, also requires filesystem unmounted
sudo xfs_repair -n /dev/mapper/vg-xfsdata

-n

/ -f -n are for read-only checks; they do not mean you can run them on any production device. Before actual repair, preserve diagnostic output, verify backup restorability, and prepare for extended downtime.

Build a Judgment Chain That Doesn't Go Astray

Why df , du , and Application Perspective Often Disagree

df

reads allocated and available blocks from the filesystem superblock, answering from the whole filesystem perspective; du traverses visible directory entries and sums file usage, answering from the current namespace and permission perspective. Differences between them do not automatically indicate filesystem corruption. Deleted-but-open files count only in df; files hidden by other mounts are invisible in the current directory tree; sparse file logical size differs from actual block count; snapshots and CoW shared blocks cannot be accurately expressed by simple directory summation.

For XFS, Btrfs, etc., metadata also consumes space. A filesystem may appear to have data space, but specific allocation groups, metadata block groups, or devices may face local pressure. Do not substitute "total remaining percentage" for filesystem-specific status. XFS can first view overall and allocation group info; Btrfs must distinguish device usage from filesystem usage:

fstype=$(findmnt -no FSTYPE -T /var/lib/myapp)
case "$fstype" in
  xfs)
    xfs_info /var/lib/myapp
    sudo xfs_db -r -c 'freesp -s' "$(findmnt -no SOURCE -T /var/lib/myapp)"
    ;;
  btrfs)
    sudo btrfs filesystem usage -T /var/lib/myapp
    sudo btrfs device usage /var/lib/myapp
    ;;
esac

xfs_db -r

uses read-only mode but still confirm the argument is an XFS device. Btrfs Data, Metadata, and System block groups allocate separately; device not full does not mean metadata has free chunks. Btrfs qgroups may also let subvolumes hit referenced space limits, manifesting as overall space available but local writes failing.

sudo btrfs quota status /var/lib/myapp
sudo btrfs qgroup show -reF /var/lib/myapp
sudo btrfs subvolume show /var/lib/myapp

Do not run btrfs balance arbitrarily at the incident scene. Balance rewrites large amounts of blocks; when space is extremely tight or devices unhealthy, it may increase I/O pressure. First read usage layout, confirm backups and device health, then use targeted filters for data or metadata, and validate cancel/recovery procedures in a test environment.

Inode Problems Must Be Solved at the Production Level

ext4 inode count is usually fixed at filesystem creation and cannot be increased online like regular capacity. The long-term solution is not daily batch deletion but changing small-file production patterns or rebuilding the filesystem: move massive sessions into a TTL-enabled database or cache, merge small objects into combined storage, limit task output counts, set lifecycle cleanup, and alert on creation rate. If ext4 rebuild is mandatory, choose appropriate bytes-per-inode on the new volume based on average file size, complete data verification and cutover before reclaiming the old volume.

Migration is high-risk. At minimum requires dual verification of capacity and inodes, application write-quiesce or incremental sync mechanism, preservation of permissions/ACLs/xattrs, checksum comparison, mount parameter recording, and a quick fallback plan to the old volume. Simply using cp -r often loses hard links, ACLs, extended attributes, or sparseness. Proper tools include validated rsync, storage snapshot replication, or application-native backup/restore, chosen per data semantics.

Logs and temporary files should be handed to explicit lifecycle management, not rely on on-call manual deletion. When checking journald usage, first view statistics, then decide vacuum per organizational retention:

journalctl --disk-usage
systemd-analyze cat-config systemd/journald.conf \
  | grep -E '^(SystemMaxUse|SystemKeepFree|RuntimeMaxUse|MaxRetentionSec)='

# Only estimate archive logs releasable by time-based cleanup, do not delete
journalctl --vacuum-time=14d --dry-run 2>/dev/null || true

Not all systemd versions support vacuum dry-run; unsupported versions will fail, the || true here is only for compatibility check. Actual vacuum irreversibly deletes archived logs; must first satisfy audit retention, confirm logs are centrally collected, and record deletion scope. Application logs should check logrotate frequency, rotate, maxsize, compression, and reopen behavior to avoid "rotation succeeded but process keeps writing old file".

Why Atomic Writes Need More Space Than They Appear

Many applications update config or database fragments not by in-place overwrite but by writing a temporary file in the same directory, fsync, then rename over the old file. This guarantees power-loss consistency but means briefly needing space for both old and new versions plus an extra inode. Compression, backup, database compaction, container image unpacking have similar peaks. Capacity planning cannot rely on steady-state file sizes alone; must account for peaks during releases, backups, restores, and compaction.

Directories themselves grow. Millions of directory entries increase lookup and metadata update cost even if file contents are tiny. Hashing files into multi-level directories improves single-directory operations but does not reduce total inode count; if application deletion speed lags creation speed, sharding only postpones failure.

Capacity Alerts Should Cover "How Long Until Exhaustion"

Fixed thresholds only detect already near-exhaustion states. More practical monitoring includes current watermark, growth rate, and predicted time to exhaustion. Compute separately for data blocks and inodes, using sufficiently long observation windows during periodic jobs like backups and batch processing. Growth is not linear; predictions should only provide lead time, not replace hard thresholds.

predict_linear(
  node_filesystem_avail_bytes{mountpoint="/var/lib/myapp"}[6h],
  24 * 3600
) < 0

predict_linear(
  node_filesystem_files_free{mountpoint="/var/lib/myapp"}[6h],
  24 * 3600
) < 0

Prediction window should cover typical workload cycles; systems with daily cycles may false-alert with 6-hour linear prediction. Alert annotations must include recent growth rate, target mount point, filesystem type, largest directory, and responsible service so on-call personnel don't have to rediscover context.

Core checks can be converged into a single read-only script. It does not perform auto-remediation because handling risks differ greatly across root causes.

#!/usr/bin/env bash
set -Eeuo pipefail

target=${1:-}
if [[ -z "$target" || ! -e "$target" ]]; then
  echo "Usage: $0 <existing file or directory>" >&2
  exit 2
fi

echo '== identity =='
date -Is
id

echo '== mount =='
findmnt -T "$target" -o TARGET,SOURCE,FSTYPE,OPTIONS,SIZE,USED,AVAIL

echo '== blocks =='
df -hT "$target"

echo '== inodes =='
df -i "$target"

echo '== path =='
namei -l "$target"

echo '== deleted-open files on same host =='
lsof +L1 2>/dev/null | head -n 20 || true

echo '== recent kernel storage messages =='
journalctl -k --since '-30 min' --no-pager 2>/dev/null \
  | grep -Ei 'enospc|no space|quota|ext4|xfs|btrfs|i/o error|read-only' \
  | tail -n 50 || true

Save the script as enospc-check.sh and run with the actual target directory. lsof may have overhead on busy hosts; if strict control is needed, remove that segment and run separately during maintenance.

Post-Fix Verification Must Go Beyond df

After root cause remediation, verify create, write, sync, and delete as the application identity, while confirming error rates, logs, and resource levels recover. Whether a system-wide sync syscall is needed depends on the application; below uses Python to fsync a single probe file without triggering full-system sync.

sudo -u myapp python3 - <<'PY'
import os
from pathlib import Path

p = Path('/var/lib/myapp/.write-probe')
try:
    with p.open('wb') as f:
        f.write(b'enospc verification
')
        f.flush()
        os.fsync(f.fileno())
    print(p.stat())
finally:
    p.unlink(missing_ok=True)
PY

Then re-check blocks, inodes, quotas, and service logs, and observe at least one normal business cycle.

df -hT /var/lib/myapp
df -i /var/lib/myapp
sudo quota -vs myapp 2>/dev/null || true
journalctl -u myapp.service --since '-10 min' --no-pager \
  | grep -Ei 'enospc|no space|quota|write|error' || true

Monitoring should at minimum cover both data blocks and inodes. With node_exporter, compute available bytes ratio and inode free ratio separately, excluding read-only and pseudo filesystems.

100 * node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
  / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}

100 * node_filesystem_files_free{fstype!~"tmpfs|overlay|squashfs"}
  / node_filesystem_files{fstype!~"tmpfs|overlay|squashfs"}

Alerts must correlate by instance, device, mountpoint, and fstype, not just hostname. Also combine quota metrics, container ephemeral-storage, write error rates, and capacity growth velocity to avoid the "disk clearly has 40%" illusion again.

Final Judgment Summary

Symptom: df -h has space, df -i 100% — Primary Evidence: Inode usage, hotspot directory file count — Common Remediation: Cleanup small files by lifecycle, fix generation strategy

Symptom: Root partition has space, target still ENOSPC — Primary Evidence: findmnt -T, process mount namespace — Common Remediation: Address the real target mount point

Symptom: Root writable, application user not writable — Primary Evidence: Quota, ext4 reserved blocks, application identity probe — Common Remediation: Adjust quota or release that user's usage

Symptom: du small, df large — Primary Evidence: lsof +L1 — Common Remediation: Let process close or reopen file normally

Symptom: Host disk has space, container fails — Primary Evidence: Overlay, data-root, ephemeral-storage — Common Remediation: Clean or expand correct container storage layer

Symptom: Kernel shows filesystem errors — Primary Evidence: Kernel journal, mount options — Common Remediation: Stop writes, snapshot, offline check and repair

The correct question for ENOSPC is not "which disk is full" but "in what namespace, on what filesystem, under what identity, did this write request which kind of limited resource". Once these four dimensions are confirmed, the 40% remaining space is no longer a contradiction but merely an unrelated or incomplete metric.