Enable GPU Acceleration in Docker and Kubernetes with NVIDIA Toolkit

This guide shows how to enable GPU support in Docker and Kubernetes using the NVIDIA Container Toolkit.

Check System Environment

Run commands to view OS version and Kubernetes client/server versions.

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
# cat /etc/redhat-release
Rocky Linux release 9.3 (Blue Onyx)

# kubectl version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.25.16
WARNING: version difference between client (1.30) and server (1.25) exceeds the supported minor version skew of +/-1

Install NVIDIA Docker Plugin

Add the NVIDIA repository, enable experimental packages, and install the toolkit.

# curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
# sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list
# sudo apt-get update
# sudo apt-get install -y nvidia-container-toolkit

Configure Docker to Use NVIDIA Runtime

Run the configuration command and restart Docker.

# sudo nvidia-ctk runtime configure --runtime=docker
INFO[0000] Loading config from /etc/docker/daemon.json
INFO[0000] Wrote updated config to /etc/docker/daemon.json
INFO[0000] It is recommended that docker daemon be restarted.
# systemctl daemon-reload
# systemctl restart docker

Resulting /etc/docker/daemon.json should contain a runtimes entry for nvidia:

{
  "runtimes": {
    "nvidia": {
      "path": "nvidia-container-runtime",
      "args": []
    }
  }
}

Verify GPU Access from Docker

Run a test container.

# docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

The output shows GPU model, temperature, power usage, and memory, confirming successful GPU access.

Deploy NVIDIA Device Plugin in Kubernetes

Create the DaemonSet.

$ kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.16.1/deployments/static/nvidia-device-plugin.yml

Key parts of the DaemonSet manifest:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      containers:
      - name: nvidia-device-plugin-ctr
        image: nvcr.io/nvidia/k8s-device-plugin:v0.16.1
        env:
        - name: FAIL_ON_INIT_ERROR
          value: "false"
        securityContext:
          allowPrivilegeEscalation: false
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins

Create a GPU‑enabled Pod

Example pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: ffmpeg-pod
spec:
  nodeSelector:
    gpu: "true"
  containers:
  - name: ffmpeg-container
    image: nightseas/ffmpeg:latest
    command: ["/bin/bash","-c","tail -f /dev/null"]
    resources:
      limits:
        nvidia.com/gpu: 1

Label a node so the DaemonSet and pod run on GPU‑capable nodes.

# kubectl label nodes aiserver003087 gpu=true

Test GPU Workload with FFmpeg

Copy a video into the pod and run an FFmpeg command that uses CUDA acceleration.

# kubectl cp test.mp4 ffmpeg-pod:/root
# kubectl exec -it ffmpeg-pod bash
# ffmpeg -hwaccel cuvid -c:v h264_cuvid -i test.mp4 -vf scale_npp=1280:720 -vcodec h264_nvenc out.mp4

Successful creation of out.mp4 confirms GPU usage.

Optional: Pin Specific GPU Devices

When multiple GPUs are present, set the CUDA_VISIBLE_DEVICES environment variable or use the nvidia.com/gpu resource selector to target a particular device.

Following these steps provides a reproducible workflow for enabling GPU acceleration in Docker containers and Kubernetes workloads.