Ubuntu Auto-Update Breaks NVIDIA Drivers: Fixing nvidia-smi & K8s GPU Plugin Failures

Fault Phenomenon

Developers reported GPU node issues. Running nvidia-smi returned:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.

In the Kubernetes cluster, GPU-related pods such as nvidia-device-plugin entered RunContainerError with the event:

failed to initialize NVML: Driver Not Loaded

The driver was gone; GPUs became unusable.

Root Cause

Investigation revealed Ubuntu's unattended-upgrades had automatically upgraded the kernel overnight. unattended-upgrades is the official Ubuntu/Debian unattended upgrade tool that installs security updates without user intervention. While it improves security, it can break kernel-dependent modules.

NVIDIA drivers are kernel modules ( .ko files) compiled against a specific kernel version. When the kernel changes, the old modules are incompatible with the new kernel. After reboot, the system loads the new kernel, but the corresponding NVIDIA modules are missing from the new kernel's module directory, so the driver fails to load.

Three-Step Fix

Confirm the Cause

On the affected node, run:

# Check current running kernel version
uname -r

# Check if NVIDIA driver modules exist for current kernel
ls /lib/modules/$(uname -r)/kernel/drivers/video/nvidia*

If the ls command returns No such file or directory , the kernel upgrade caused the driver module loss.

Repair Driver (Recommended DKMS Method)

Because driver sources may have been cleaned during the update, reinstall the DKMS package to rebuild modules for the current kernel:

# 1. Install kernel headers for current kernel (required to compile modules)
sudo apt update
sudo apt install linux-headers-$(uname -r)

# 2. Reinstall NVIDIA DKMS package (system auto-compiles driver for current kernel)
# Note: adjust version as needed, e.g., nvidia-dkms-550 or nvidia-dkms-535
sudo apt install nvidia-dkms-550

# 3. Verify driver recovery
nvidia-smi

Restore Kubernetes Cluster State

Once nvidia-smi outputs GPU information normally, the Kubernetes GPU plugin usually recovers automatically. If not, manually delete the plugin pod to trigger DaemonSet recreation:

kubectl delete pod <gpu-plugin-pod-name> -n kube-system

Prevention Measures

Production GPU nodes must not allow uncontrolled kernel upgrades. Lock the current stable kernel version:

# Lock current kernel and headers (replace version with actual running version)
sudo apt-mark hold linux-image-5.15.0-181-generic linux-headers-5.15.0-181-generic

After locking, the system will not upgrade this kernel, but other packages update normally.

To manually upgrade the kernel later, unlock first:

sudo apt-mark unhold linux-image-xxx linux-headers-xxx

After upgrading, reinstall the corresponding nvidia-dkms package, then lock the new kernel version.

Conclusion

The issue is straightforward but insidious because it happens silently overnight. By the time it's discovered, business may have been interrupted for hours. Rather than firefighting afterward, lock the kernel in advance. On GPU nodes, kernel stability matters more than novelty.