Mastering NFS: A Complete Guide to Setup, Troubleshooting, and Performance Optimization

1. Overview

NFS (Network File System) is the most widely used network file sharing protocol in UNIX/Linux environments, allowing applications to access remote files transparently via standard POSIX calls. It relies on RPC for communication and has evolved through several versions since its introduction in 1984.

1.1 NFS Versions

NFSv3 (RFC 3530, 1995) : Stateless, uses multiple ports (mountd, nlockmgr, statd), supports asynchronous writes, 64‑byte file handles, still dominant for high‑performance computing.

NFSv4 (RFC 7530, 2003) : Stateful, introduces leases and delegations, consolidates all traffic on TCP port 2049, adds compound operations, supports ACLs, mandatory locking, and Kerberos authentication.

NFSv4.1 (RFC 8881, 2010) : Adds session support and pNFS for parallel I/O.

NFSv4.2 (RFC 7862, 2016) : Adds server‑side copy, sparse file support, I/O advice, and SELinux‑labeled NFS.

1.2 Mount Process (NFSv3 example)

1. Client queries rpcbind (port 111) for mountd port
2. Client sends MOUNT request to mountd with export path
3. Server checks /etc/exports and validates client IP
4. Server returns a file handle
5. Client uses the file handle on NFS port 2049 for subsequent operations
6. File locks are managed by nlockmgr (separate port)
7. After a server reboot, lock recovery is coordinated via statd

1.3 Common Failure Categories

Network connectivity issues (blocked ports, firewall rules).

Server configuration errors (exports syntax, services not started).

Authentication and permission problems (root_squash, UID/GID mismatches, Kerberos misconfiguration).

Version or parameter incompatibilities (unsupported NFS version, mismatched mount options).

Runtime faults (stale file handles, server unresponsiveness, network jitter).

2. Core Concepts

RPC & Port Mapping : NFSv3 uses multiple daemon processes (rpcbind, mountd, nlockmgr, statd) each listening on dynamic ports; fixing these ports simplifies firewall rules.

File Handle : NFS identifies files by opaque binary handles generated by the server; if the underlying inode changes, the handle becomes stale, causing ESTALE errors.

root_squash : By default, NFS maps the client’s root UID (0) to the anonymous user (nobody) for security; this can cause write permission failures unless adjusted.

3. Environment Requirements

Operating System: Ubuntu 24.04 LTS or Rocky Linux 9.5 (or other recent LTS distributions).

Kernel: 6.12+ (includes latest NFS client/server improvements).

Packages: nfs-common (client), nfs-kernel-server (Ubuntu) or nfs-utils (Rocky).

Utilities for troubleshooting: tcpdump, wireshark-cli, Prometheus node_exporter (with --collector.nfs).

4. Detailed Steps

4.1 Preparation

4.1.1 Install NFS Tools

Ubuntu client:

# Install NFS client tools
sudo apt update
sudo apt install -y nfs-common
# nfs-common provides mount.nfs, showmount, nfsstat, rpcinfo

Rocky Linux client:

# Install NFS client tools
sudo dnf install -y nfs-utils
# Enable rpcbind for NFSv3
sudo systemctl enable --now rpcbind

Ubuntu server:

# Install NFS server
sudo apt install -y nfs-kernel-server
sudo systemctl enable --now nfs-kernel-server

Rocky Linux server:

# Install NFS server
sudo dnf install -y nfs-utils
sudo systemctl enable --now nfs-server

4.2 Server Export Configuration

Create shared directories and set permissions:

sudo mkdir -p /data/shared /data/readonly /data/app
sudo chown nobody:nogroup /data/shared || sudo chown nobody:nobody /data/shared
sudo chmod 755 /data/shared /data/readonly
sudo chown 1000:1000 /data/app
sudo chmod 775 /data/app

Define exports (ensure no spaces between client spec and options):

/data/shared    10.0.0.0/24(rw,sync,no_subtree_check,no_root_squash)
/data/readonly  10.0.0.0/24(ro,sync,no_subtree_check)
/data/app       10.0.0.0/24(rw,sync,no_subtree_check,all_squash,anonuid=1000,anongid=1000)

Fix dynamic ports for NFSv3 auxiliary services (mountd, nlockmgr, statd) in /etc/nfs.conf:

[mountd]
port=20048
[statd]
port=32765
[lockd]
port=32803
udp-port=32803

Reload configuration and verify:

sudo systemctl restart rpcbind nfs-server
sudo exportfs -ra
sudo exportfs -v
rpcinfo -p localhost

4.3 Client Mounting and Verification

Basic mount (NFSv3):

sudo mount -t nfs nfs-server:/data/shared /mnt/nfs

Specify version and performance options (example for large files):

sudo mount -t nfs -o vers=4.2,rsize=1048576,wsize=1048576,hard,noatime,proto=tcp nfs-server:/data/shared /mnt/nfs

Verify mount:

df -h /mnt/nfs
mount | grep /mnt/nfs

4.4 Permission and Authentication Checks

Check UID/GID mapping consistency between client and server.

Inspect root_squash settings; use no_root_squash only after risk assessment.

For NFSv4, ensure /etc/idmapd.conf Domain values match on both sides.

If Kerberos is used, verify rpc-gssd service, keytab, and ticket validity.

4.5 Performance Tuning

Key mount parameters: rsize / wsize: default 1 MiB; increase for large sequential I/O. hard / soft: use hard in production for data integrity. noatime / relatime: reduce metadata writes. nconnect=N (Linux 5.3+): open multiple TCP connections to overcome single‑connection bandwidth limits.

4.6 Automated Recovery Script (example)

#!/bin/bash
# nfs_health_check.sh – monitors mount points and attempts recovery
MOUNT_POINTS="/mnt/nfs /mnt/data"
CHECK_TIMEOUT=10
WEBHOOK_URL="https://example.com/webhook"
MAX_RECOVERY_ATTEMPTS=3
RECOVERY_STATE_DIR="/tmp/nfs_recovery_state"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
# Functions omitted for brevity – see source for send_alert, check_mount_exists, attempt_recovery, etc.
for mp in $MOUNT_POINTS; do
  # Verify existence, responsiveness, writability, and attempt auto‑recovery when needed.
  ...
done

5. Best Practices and Caveats

5.1 Server Hardening

Export only to specific IP ranges; avoid * wildcards.

Prefer sync over async for data safety.

Always use no_subtree_check to improve performance.

Adjust nfsd thread count based on concurrent client load (e.g., 32 threads for 16 clients).

Disable unused protocol versions (e.g., set vers3=n if not needed).

5.2 Mount Parameter Templates

# Common options
COMMON_OPTS="hard,noatime,_netdev,noresvport"
# Large‑file workloads
LARGE_OPTS="$COMMON_OPTS,rsize=1048576,wsize=1048576,nconnect=4"
# Small‑file workloads
SMALL_OPTS="$COMMON_OPTS,rsize=65536,wsize=65536,lookupcache=all"
# Database storage (strong consistency)
DB_OPTS="$COMMON_OPTS,sync,lookupcache=pos"
# Kubernetes PV backend
K8S_OPTS="$COMMON_OPTS,noresvport,rsize=1048576,wsize=1048576"

5.3 High‑Availability Options

Use autofs for on‑demand mounting and automatic unmount after idle timeout.

Deploy NFS‑Ganesha or a clustered NFS service for load balancing and failover.

Consider NFS over RDMA for ultra‑low latency in InfiniBand or RoCE networks.

6. Troubleshooting and Monitoring

6.1 Log Inspection

# Server logs
sudo journalctl -u nfs-server -f
# rpcbind logs
sudo journalctl -u rpcbind --since "1 hour ago"
# Kernel NFS messages
dmesg | grep -i nfs
# Enable client debug (temporary)
sudo rpcdebug -m nfs -s all
# Disable debug after investigation
sudo rpcdebug -m nfs -c all

6.2 Common Issues

Connection timed out : network or firewall block (port 111 for NFSv3, port 2049 for NFSv4).

Access denied : client IP not permitted in /etc/exports.

Permission denied (write) : root_squash or directory permissions.

Stale file handle : server-side file recreation; unmount and remount.

All files show as nobody : mismatched NFSv4 idmapping domain.

Hard mount causing D‑state processes : server unresponsive; use monitoring to detect and umount -f or umount -l as needed.

6.3 Performance Monitoring

Collect NFS metrics with node_exporter (enable --collector.nfs and --collector.nfsd) and monitor key counters such as node_nfs_rpc_retransmissions_total, node_nfsd_server_threads, and latency statistics from /proc/self/mountstats. Example Prometheus alerts are provided for high retransmission rates, exhausted server threads, and unresponsive mount points.

7. Backup and Recovery

7.1 Configuration Backup Script

#!/bin/bash
# nfs_config_backup.sh – backs up server and client NFS configuration
BACKUP_DIR="/backup/nfs/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"
# Server files
if systemctl is-active --quiet nfs-server || systemctl is-active --quiet nfs-kernel-server; then
  cp /etc/exports "$BACKUP_DIR/" 2>/dev/null
  cp /etc/nfs.conf "$BACKUP_DIR/" 2>/dev/null
  cp -r /etc/exports.d "$BACKUP_DIR/" 2>/dev/null
  sudo exportfs -v > "$BACKUP_DIR/active_exports.txt"
fi
# Client files
grep -E "nfs|nfs4" /etc/fstab > "$BACKUP_DIR/fstab_nfs_entries.txt" 2>/dev/null
mount -t nfs,nfs4 > "$BACKUP_DIR/current_mounts.txt" 2>/dev/null
cp /etc/idmapd.conf "$BACKUP_DIR/" 2>/dev/null
# Systemd mount/automount units
find /etc/systemd/system/ -name "*.mount" -o -name "*.automount" -exec cp {} "$BACKUP_DIR/" \; 2>/dev/null
# Autofs configuration
cp /etc/auto.master "$BACKUP_DIR/" 2>/dev/null
cp /etc/auto.nfs "$BACKUP_DIR/" 2>/dev/null
# Archive
tar czf "${BACKUP_DIR}.tar.gz" -C "$(dirname $BACKUP_DIR)" "$(basename $BACKUP_DIR)"
rm -rf "$BACKUP_DIR"
echo "NFS configuration backed up to ${BACKUP_DIR}.tar.gz"

7.2 Restoration Procedure

# 1. Extract backup
sudo tar xzf /backup/nfs/20260313_143000.tar.gz -C /tmp/
# 2. Restore server configuration
sudo cp /tmp/20260313_143000/exports /etc/exports
sudo cp /tmp/20260313_143000/nfs.conf /etc/nfs.conf
sudo systemctl restart nfs-server
sudo exportfs -v
# 3. Restore client entries
sudo cp /tmp/20260313_143000/fstab_nfs_entries.txt /etc/fstab
sudo mount -a
# 4. Verify
df -h -t nfs4

8. Summary

This guide covered NFS fundamentals, version differences, detailed installation steps for both Ubuntu and Rocky Linux, common failure categories, core concepts such as RPC, file handles, and root_squash, performance‑tuning parameters, real‑world case studies, automated health‑check scripting, best‑practice hardening, monitoring with Prometheus, and backup/recovery procedures. Following these recommendations ensures reliable, secure, and high‑performance NFS deployments.

9. References

Linux NFS Wiki – https://linux-nfs.org/wiki/index.php/Main_Page

nfs‑utils source and documentation – https://git.linux-nfs.org/?p=steved/nfs-utils.git

RFC 7530 – NFS Version 4 Protocol

RFC 8881 – NFS Version 4.1 Protocol

RFC 7862 – NFS Version 4.2 Protocol

Red Hat NFS Administration Guide – https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/managing_file_systems/

man 5 exports, man 5 nfs, man 8 mount.nfs