Systematic Root‑Cause Analysis for NFS Mount Failures

Overview

NFS (Network File System) is the most widely used network‑file sharing protocol on Linux. The kernel provides native support, so a client can mount a remote export with a single mount command.

Version comparison

NFSv3 – stateless, dynamic ports (requires portmapper), AUTH_SYS authentication, separate NLM lock service, firewall‑unfriendly, good performance, weak security.

NFSv4 – stateful, fixed TCP port 2049, AUTH_SYS + Kerberos optional, built‑in locking, firewall‑friendly, improved performance, strong security.

NFSv4.1 – adds session support and parallel I/O (pNFS).

NFSv4.2 – adds server‑side copy and other extensions.

Production‑environment recommendations

Trusted internal network: use NFSv4 (single port, simple configuration).

Kerberos authentication required: use NFSv4.1+.

Server‑side copy or sparse files needed: use NFSv4.2.

Compatibility with legacy systems: use NFSv3.

Architecture layers

+----------------------------------+
|  Application layer: file I/O      |
+----------------------------------+
|  NFS protocol layer: RPC wrapper |
|  - NFSv3: MOUNT + NFS + NLM       |
|  - NFSv4: unified protocol        |
+----------------------------------+
|  Authentication layer: AUTH_SYS / Kerberos |
|  - UID/GID mapping               |
|  - idmapd (NFSv4)                |
+----------------------------------+
|  Transport layer: TCP/UDP        |
|  - Port 2049 (NFS)               |
|  - Port 111 (portmapper, v3)     |
+----------------------------------+
|  Network layer: IP routing / firewall |
|  - iptables/nftables/firewalld   |
|  - Security groups / ACLs        |
+----------------------------------+

When troubleshooting, start from the bottom: verify network connectivity, then port reachability, then authentication, and finally protocol‑level issues.

Key daemons and services

rpcbind

– RPC port mapper (required for NFSv3). rpc.mountd – handles MOUNT requests (required for NFSv3). rpc.nfsd – serves NFS requests (required for both v3 and v4). rpc.statd – lock recovery (required for NFSv3, not for v4). rpc.lockd – file lock management (kernel thread for v3). rpc.idmapd – UID/GID ↔ username mapping (required for NFSv4). rpc.gssd / rpc.svcgssd – Kerberos authentication (optional, required when sec=krb5* is used).

A showmount timeout usually indicates a problem with rpcbind or rpc.mountd.

RPC mechanism and port mapping (NFSv3)

Client                               Server
 |                                    |
 | 1. Query portmapper (port 111)      |
 |    → "Which port does NFS run on?" |
 |    ← "On port 2049"                |
 |                                    |
 | 2. Query mountd port                |
 |    → "Which port is mountd?"        |
 |    ← "On port 20048"               |
 |                                    |
 | 3. MOUNT request (port 20048)      |
 |    → "Mount /data/share"           |
 |    ← "Allowed, here is the file handle" |
 |                                    |
 | 4. NFS operations (port 2049)      |
 |    → read/write/lookup …            |

NFSv4 simplifies the flow: all operations use TCP port 2049, eliminating the need for portmapper and mountd.

Full mount procedure

mount -t nfs server:/share /mnt

1. DNS resolves the server IP.
2. TCP connects to port 2049 (or first queries portmapper for NFSv3).
3. Sends a MOUNT request.
4. Server checks <code>/etc/exports</code> for permission.
5. Server checks whether the client IP is allowed.
6. Server returns a file handle.
7. Client uses the file handle for subsequent NFS operations.
8. Kernel mounts the filesystem at <code>/mnt</code>.

Any step that fails results in a mount failure; error messages are often cryptic.

Common error messages

mount.nfs: Connection timed out

– firewall block or network unreachable. mount.nfs: access denied by server – incorrect /etc/exports configuration. mount.nfs: No such file or directory – wrong export path on the server. mount.nfs: Permission denied – root_squash or authentication problem. mount.nfs: Stale file handle – server reboot or export change. mount.nfs: Network is unreachable – routing or network configuration issue. mount.nfs: Protocol not supported – NFS version mismatch.

Layered troubleshooting model (bottom‑up)

1️⃣ Network connectivity
   └→ ping / traceroute / mtr
   └→ check routing

2️⃣ Port reachability
   └→ nc -zv server 2049
   └→ nc -zv server 111 (NFSv3)
   └→ firewall rules

3️⃣ RPC service status
   └→ rpcinfo -p server
   └→ showmount -e server
   └→ daemon status

4️⃣ Exports configuration
   └→ exportfs -v
   └→ IP matching rules
   └→ option correctness (watch out for spaces)

5️⃣ Permissions & authentication
   └→ root_squash / all_squash
   └→ UID/GID consistency
   └→ idmapd Domain (NFSv4)
   └→ Kerberos tickets (if enabled)

6️⃣ Protocol & version
   └→ NFS version negotiation
   └→ mount options compatibility
   └→ kernel module loading

Following this order avoids wasting time on higher‑level symptoms when a lower layer is already broken.

Applicable scenarios

New NFS deployment that fails to mount.

Server reboot causing client mount errors.

Network changes that break NFS access.

Read/write permission problems after mounting.

Performance degradation investigations.

Migrating from NFSv3 to NFSv4.

Environment requirements

Operating System: Ubuntu 24.04 LTS or Rocky Linux 9.5 (kernel 6.12+). nfs-utils ≥ 2.7 (client and server utilities). rpcbind ≥ 1.2 (required for NFSv3). nfs-kernel-server ≥ 2.7 (Ubuntu server package).

# Ubuntu 24.04 installation
sudo apt install -y nfs-kernel-server nfs-common

# Rocky Linux 9.5 installation
sudo dnf install -y nfs-utils

# Verify versions
nfsstat --version 2>/dev/null || rpcinfo -p localhost | head -5

Detailed steps

2.1 NFS server checks

Check NFS service status

# Verify NFS service is running
sudo systemctl status nfs-server   # Rocky
sudo systemctl status nfs-kernel-server   # Ubuntu

# Check related services
sudo systemctl status rpcbind
sudo systemctl status nfs-mountd
sudo systemctl status nfs-idmapd

# Show server statistics
nfsstat -s

Check /etc/exports configuration

# View current export file
cat /etc/exports

# Show what is actually exported (may differ from the file)
sudo exportfs -v

# Common export errors (illustrated with comments)
# Error 1: Space between IP and parentheses – WRONG
#   /data/share 192.168.1.0/24 (rw,sync)
# Correct form – NO SPACE
#   /data/share 192.168.1.0/24(rw,sync)

# Error 2: Directory does not exist
#   /data/nonexistent 192.168.1.0/24(rw,sync)

# Error 3: Wrong netmask (too broad)
#   /data/share 192.168.1.0/16(rw,sync)

Check NFS ports

# List listening ports for NFS services
ss -tlnp | grep -E ":(2049|111|20048) "

# Show RPC‑registered services (needed for NFSv3)
rpcinfo -p localhost

Fix dynamic ports for NFSv3 (firewall‑friendly)

# Fix ports in /etc/nfs.conf (Ubuntu/Rocky)
cat >> /etc/nfs.conf <<'EOF'
[mountd]
port = 20048

[statd]
port = 32765
outgoing-port = 32766

[lockd]
port = 32803
udp-port = 32803
EOF

# Restart services
sudo systemctl restart nfs-server
sudo systemctl restart rpc-statd

2.2 NFS client checks

Inspect mount options

# Show current NFS mounts
mount -t nfs,nfs4

# Detailed mount options for a specific mount point
nfsstat -m

Test manual mount (debug mode)

# Verbose mount for troubleshooting
sudo mount -v -t nfs 192.168.1.100:/data/share /mnt/debug

# Force a specific version
sudo mount -t nfs -o vers=4 192.168.1.100:/data/share /mnt/debug
sudo mount -t nfs -o vers=3 192.168.1.100:/data/share /mnt/debug

Kernel module verification

# Verify NFS kernel modules are loaded
lsmod | grep nfs
# Load if missing
sudo modprobe nfs
sudo modprobe nfsv4

2.3 Network connectivity checks

# Basic ping test
ping -c 3 192.168.1.100

# Traceroute if ping fails
traceroute -n 192.168.1.100
mtr -n --report 192.168.1.100

# TCP port test (NFSv4)
nc -zv -w 5 192.168.1.100 2049
# Portmapper (NFSv3)
nc -zv -w 5 192.168.1.100 111
# Mountd (NFSv3)
nc -zv -w 5 192.168.1.100 20048

# Showmount test (requires portmapper)
showmount -e 192.168.1.100

2.4 NFSv4‑specific issues

idmapd configuration

# Check idmapd.conf
cat /etc/idmapd.conf
# Example entry
# [General]
# Domain = example.com   # Must be identical on client and server

# Verify idmapd is running
sudo systemctl status nfs-idmapd

# Fix mismatched domain
sudo sed -i 's/^#Domain = .*/Domain = example.com/' /etc/idmapd.conf
sudo systemctl restart nfs-idmapd

# Clear idmap cache
sudo nfsidmap -c

NFSv4 pseudo‑filesystem

# Export with pseudo‑root (fsid=0)
# /etc/exports
/data/share 192.168.1.0/24(rw,sync,fsid=0)

# Client mount using the pseudo‑root
sudo mount -t nfs -o vers=4 192.168.1.100:/ /mnt
# If this fails but NFSv3 works, check the fsid setting.

2.5 Kerberos authentication checks

# Verify Kerberos tickets
klist
# Obtain a ticket if missing
kinit [email protected]

# Check keytab entries
klist -kt /etc/krb5.keytab
# Ensure an entry for nfs/hostname@REALM exists

# Verify Kerberos daemons
sudo systemctl status rpc-gssd   # client
sudo systemctl status rpc-svcgssd   # server

# Review /etc/krb5.conf (sections: [libdefaults], [realms])
cat /etc/krb5.conf

# Common Kerberos errors and fixes
# "Server not found in Kerberos database" → missing nfs/hostname entry in keytab
# "GSS‑API error: No credentials were supplied" → rpc-gssd not running or wrong keytab path

# DNS reverse lookup must be correct for Kerberos
host 192.168.1.100
nslookup server.example.com

2.6 Performance troubleshooting

Inspect mount options

# Show current mount options
nfsstat -m
# Example output (truncated)
# /mnt/share from 192.168.1.100:/data/share
#   Flags: rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,
#          hard,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none

# Important parameters
# rsize/wsize – read/write block size (default 1 MiB, range 4 KiB‑1 MiB)
# timeo – RPC timeout (0.1 s units, default 60 s)
# retrans – number of retries (default 2)
# proto – transport protocol (NFSv4 forces TCP)
# hard/soft – hard retries indefinitely, soft returns error after timeout

Read/Write performance tests

# Direct‑IO write test (bypass cache)
dd if=/dev/zero of=/mnt/share/testfile bs=1M count=1024 oflag=direct
# Expected: ~87 MB/s

# Direct‑IO read test
dd if=/mnt/share/testfile of=/dev/null bs=1M count=1024 iflag=direct
# Expected: ~125 MB/s

# More precise test with fio
fio --name=nfs_test --directory=/mnt/share --rw=randwrite \
    --bs=4k --size=1G --numjobs=4 --time_based --runtime=60 \
    --group_reporting --direct=1

# Small‑file test
fio --name=smallfile --directory=/mnt/share --rw=write \
    --bs=4k --size=4k --nrfiles=10000 --numjobs=1 --direct=1

Performance bottleneck identification

# Client RPC statistics
nfsstat -c
# Example line: calls 125643 retrans 23 authrefrsh 125643
# High retrans count → possible network issue.

# Server statistics
nfsstat -s

# NFS I/O statistics per second
nfsiostat 1
# Columns: ops/s, rpc, bklog, read/write ops, kB/s, avg RTT, avg execution time

# TCP connection statistics
ss -s
# High TIME_WAIT or retransmit counts suggest network problems.

Mount option tuning examples

# High‑performance (large files)
192.168.1.100:/data/share /mnt/share nfs4 \
    rsize=1048576,wsize=1048576,hard,intr,tcp,noatime,_netdev 0 0

# High‑reliability (data safety)
192.168.1.100:/data/share /mnt/share nfs4 \
    rsize=1048576,wsize=1048576,hard,sync,timeo=600,retrans=5,_netdev 0 0

# Web application (read‑heavy)
192.168.1.100:/data/www /mnt/www nfs4 \
    rsize=1048576,wsize=1048576,hard,intr,noatime,_netdev 0 0

2.7 Verify mount success (script)

#!/bin/bash
SERVER="192.168.1.100"
SHARE="/data/share"
MOUNT_POINT="/mnt/share"
OPTIONS="hard,intr,timeo=600,retrans=2"

# 1. Check mount point
if mountpoint -q "$MOUNT_POINT"; then
  echo "[OK] $MOUNT_POINT is mounted"
else
  echo "[FAIL] $MOUNT_POINT not mounted, attempting mount"
  sudo mount -t nfs4 -o $OPTIONS "$SERVER:$SHARE" "$MOUNT_POINT" && 
    echo "[OK] $MOUNT_POINT mounted" || echo "[FAIL] mount failed"
  exit 1
fi

# 2. Write test
TEST_FILE="$MOUNT_POINT/.nfs_test_$$"
if echo test > "$TEST_FILE"; then
  echo "[OK] Write test passed"
  rm -f "$TEST_FILE"
else
  echo "[FAIL] Write test failed"
fi

# 3. Show mount options
nfsstat -m | grep -A5 "$MOUNT_POINT" || mount | grep "$MOUNT_POINT"

Best practices and caveats

NFS server hardening

# 1. Restrict client IP ranges in /etc/exports
/data/share 192.168.1.0/24(rw,sync,no_subtree_check)

# 2. Prefer NFSv4 (single port) and disable NFSv3
# Rocky Linux
echo 'RPCNFSDARGS="-V 4 -N 3"' | sudo tee -a /etc/sysconfig/nfs
# Or in /etc/nfs.conf
# [nfsd]
# vers3 = n
# vers4 = y

# 3. Use Kerberos for high security
/data/share 192.168.1.0/24(rw,sync,sec=krb5p)

# 4. Enforce secure ports (clients must use privileged ports)
/data/share 192.168.1.0/24(rw,sync,secure)

# 5. Disable unnecessary RPC services (e.g., rpc-statd if file locking is not needed)
sudo systemctl stop rpc-statd
sudo systemctl disable rpc-statd

High‑availability deployment options

# Option A: DRBD + Pacemaker (block‑level replication, active‑passive, failover 30‑60 s)
# Option B: GlusterFS backend + NFS‑Ganesha (distributed, active‑active, horizontal scaling)
# Option C: Simple NFS + keepalived VIP (requires shared storage, easy to configure)

# Example keepalived configuration (primary node)
# /etc/keepalived/keepalived.conf
vrrp_script chk_nfs {
    script "/usr/local/bin/check_nfs.sh"
    interval 5
    weight -20
}

vrrp_instance NFS_HA {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass nfs_ha_pass
    }
    virtual_ipaddress { 192.168.1.200/24 }
    track_script { chk_nfs }
}

# Health‑check script used above
cat > /usr/local/bin/check_nfs.sh <<'EOF'
#!/bin/bash
systemctl is-active --quiet nfs-server && \
    exportfs -v | grep -q "/" && \
    ss -tlnp | grep -q ":2049 "
exit $?
EOF
chmod +x /usr/local/bin/check_nfs.sh

fstab and autofs configuration

# /etc/fstab entry (NFSv4, network‑ready, background mount)
192.168.1.100:/data/share /mnt/share nfs4 hard,intr,timeo=600,retrans=2,_netdev,bg 0 0

# Install autofs (Ubuntu/Rocky)
sudo apt install -y autofs   # Ubuntu
sudo dnf install -y autofs   # Rocky

# /etc/auto.master
/mnt  /etc/auto.nfs  --timeout=300

# /etc/auto.nfs
share   -rw,hard,intr,vers=4   192.168.1.100:/data/share
backup  -ro,hard,intr,vers=4   192.168.1.100:/data/backup

sudo systemctl enable --now autofs
# Accessing /mnt/share triggers an automatic mount; idle for 300 s causes auto‑unmount.

Fault diagnosis and monitoring

Log inspection

# Server logs
sudo journalctl -u nfs-server -n 50 --no-pager
sudo journalctl -u rpcbind -n 20 --no-pager

# Enable kernel NFS debugging (short‑term only)
echo 32767 | sudo tee /proc/sys/sunrpc/nfs_debug
# Capture NFS packets for deep analysis
sudo tcpdump -i eth0 -nn port 2049 -c 100 -w /tmp/nfs_debug.pcap

Common‑error checklist (condensed)

Connection timed out – firewall blocks required ports (2049 for v4, 111 + 20048 for v3).

Access denied – /etc/exports mis‑configuration or IP mismatch.

Permission denied – root_squash or directory permissions.

Stale file handle – server reboot or export change; fix with umount -l then remount.

Owner shows as nobody – inconsistent

idmapd.conf

Domain

between client and server.

Mount timeout (NFSv3) – portmapper/mountd ports blocked; open them or upgrade to NFSv4.

Boot‑time mount failure – network not ready; use _netdev in /etc/fstab.

Slow writes – sync mode + small I/O size; use async or increase wsize.

NFS health‑check script (Prometheus exporter)

#!/bin/bash
METRICS_FILE="/var/lib/prometheus/node-exporter/nfs.prom"

# Client RPC statistics
if [ -f /proc/net/rpc/nfs ]; then
  read_ops=$(awk '/proc4/ {print $7}' /proc/net/rpc/nfs)
  write_ops=$(awk '/proc4/ {print $8}' /proc/net/rpc/nfs)
  echo "nfs_client_read_ops_total $read_ops" > "$METRICS_FILE"
  echo "nfs_client_write_ops_total $write_ops" >> "$METRICS_FILE"
fi

# Verify each NFS mount point is reachable
mount -t nfs,nfs4 2>/dev/null | awk '{print $3}' | while read mp; do
  if mountpoint -q "$mp" && timeout 5 ls "$mp" >/dev/null 2>&1; then
    echo "nfs_mount_available{mountpoint=\"$mp\"} 1" >> "$METRICS_FILE"
  else
    echo "nfs_mount_available{mountpoint=\"$mp\"} 0" >> "$METRICS_FILE"
  fi
done

Prometheus alert rules (excerpt)

# /etc/prometheus/rules/nfs_alerts.yml
groups:
- name: nfs_alerts
  rules:
  - alert: NFSMountUnavailable
    expr: nfs_mount_available == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "NFS mount unavailable: {{ $labels.mountpoint }}"
      description: "Mount point {{ $labels.mountpoint }} has been unavailable for more than 2 minutes."

  - alert: NFSHighRetransmissions
    expr: rate(node_nfs_requests_total[5m]) > 0 and \
          rate(node_nfs_rpc_retransmissions_total[5m]) / rate(node_nfs_requests_total[5m]) > 0.05
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "NFS RPC retransmission rate high"
      description: "NFS RPC retransmission exceeds 5%, possible network issue."

Backup and recovery scripts

#!/bin/bash
# Backup NFS configuration
BACKUP_DIR="/opt/backup/nfs"
DATE=$(date +%Y%m%d)
mkdir -p "$BACKUP_DIR"

cp /etc/exports "$BACKUP_DIR/exports_$DATE"
cp /etc/nfs.conf "$BACKUP_DIR/nfs.conf_$DATE" 2>/dev/null
cp /etc/idmapd.conf "$BACKUP_DIR/idmapd.conf_$DATE" 2>/dev/null

# Export current state for reference
sudo exportfs -v > "$BACKUP_DIR/exportfs_$DATE.txt"

echo "NFS configuration backed up to $BACKUP_DIR"

Automatic remount script (cron‑based)

#!/bin/bash
LOG_FILE="/var/log/nfs_remount.log"

check_and_remount() {
  local server="$1" share="$2" mount_point="$3" options="$4"
  if ! mountpoint -q "$mount_point"; then
    echo "$(date) [WARN] $mount_point not mounted, attempting remount" >> "$LOG_FILE"
    mount -t nfs4 -o "$options" "$server:$share" "$mount_point" && \
      echo "$(date) [OK] $mount_point remounted" >> "$LOG_FILE" || \
      echo "$(date) [FAIL] $mount_point remount failed" >> "$LOG_FILE"
    return
  fi
  if ! timeout 5 ls "$mount_point" >/dev/null 2>&1; then
    echo "$(date) [WARN] $mount_point inaccessible (stale?), remounting" >> "$LOG_FILE"
    umount -l "$mount_point" 2>/dev/null
    sleep 2
    mount -t nfs4 -o "$options" "$server:$share" "$mount_point" && \
      echo "$(date) [OK] $mount_point remounted" >> "$LOG_FILE" || \
      echo "$(date) [FAIL] $mount_point remount failed" >> "$LOG_FILE"
  fi
}

# Example usage (run every 5 min via crontab)
check_and_remount "192.168.1.100" "/data/share" "/mnt/share" "hard,intr,timeo=600"

Summary

Diagnose NFS mount failures using a bottom‑up approach: network → port → RPC → exports → permissions → protocol.

NFSv4 requires only TCP 2049, simplifying firewall rules; prefer it for new deployments. root_squash is the most common source of permission problems; understand and adjust the option as needed.

Stale file handles are resolved by forced unmount ( umount -l) followed by a fresh mount; use hard,intr in /etc/fstab for reliability.

For NFSv4, the

idmapd

Domain

must match on client and server, otherwise file owners appear as nobody.

Use showmount -e to list server exports and exportfs -v to view the active export configuration.

Never leave spaces between the IP range and parentheses in /etc/exports; this is a frequent low‑level error.

Performance tuning focuses on rsize / wsize, server thread count, and network MTU/TCP parameters.

Further learning

pNFS (Parallel NFS) – NFSv4.1 parallel I/O for HPC workloads (RFC 5661, kernel docs).

NFS over RDMA – replaces TCP with RDMA for lower latency (requires InfiniBand or RoCE NICs).

NFS‑Ganesha – user‑space NFS server supporting multiple back‑ends (Ceph, GlusterFS).

CephFS vs NFS – compare distributed file system CephFS with traditional NFS for large‑scale deployments.

References

Linux NFS‑HOWTO: https://linux-nfs.org/wiki/index.php/Main_Page

Red Hat NFS Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/managing_file_systems/exporting-nfs-shares_managing-file-systems

nfs(5) man page

exports(5) man page

RFC 7530 – NFSv4 Protocol