Deploying nvidia-docker2 for GPU Workloads on Large‑Scale Kubernetes Clusters

nvidia-docker2 helps containerize legacy GPU‑accelerated applications, allocate specific GPU resources to containers, and share workloads across environments. The author records a hands‑on experience of using nvidia-docker2 in a large‑scale Kubernetes cluster.

1. Experimental Environment

CentOS Linux release 7.2.1511 (Core)

Kubernetes 1.9

GPU: NVIDIA Tesla K80

2. Installation (version 2.0)

Follow the official installation guide:

docker volume ls -q -f driver=nvidia-docker | xargs -r -I{} -n1 docker ps -q -a -f volume={} | xargs -r docker rm -f sudo yum remove nvidia-docker distribution=$(. /etc/os-release;echo $ID$VERSION_ID) curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | \ sudo tee /etc/yum.repos.d/nvidia-docker.repo sudo yum install -y nvidia-docker2 sudo pkill -SIGHUP dockerd docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi

Prerequisites include a Linux kernel >3.10, Docker >=1.12, NVIDIA GPU with architecture >Fermi, and NVIDIA drivers ~361.93.

After installation, configure Docker’s runtime by editing /etc/docker/daemon.json :

{ "default-runtime":"nvidia", "runtimes": { "nvidia": { "path": "/usr/bin/nvidia-container-runtime", "runtimeArgs": [] } } }

Restart Docker:

systemctl restart docker

3. GPU on Kubernetes

Kubernetes has supported NVIDIA GPUs since v1.6 and AMD GPUs since v1.9, but multi‑container sharing of a single GPU is still unavailable. To schedule GPUs you must enable device plugins (feature‑gates before v1.10) and install GPU drivers and the device plugin on each node.

Deploy the NVIDIA device plugin using a DaemonSet:

apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: nvidia-device-plugin-daemonset namespace: kube-system spec: template: metadata: annotations: scheduler.alpha.kubernetes.io/critical-pod: "" labels: name: nvidia-device-plugin-ds spec: tolerations: - key: CriticalAddonsOnly operator: Exists containers: - image: nvidia/k8s-device-plugin:1.9 name: nvidia-device-plugin-ctr securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] volumeMounts: - name: device-plugin mountPath: /var/lib/kubelet/device-plugins volumes: - name: device-plugin hostPath: path: /var/lib/kubelet/device-plugins

Create the DaemonSet:

kubectl create -f nvidia-docker-plugin.yml

Verify that the nvidia-device-plugin-daemonset pods are running on each GPU node.

4. Test GPU Scheduling

Deploy a test pod that requests one GPU:

apiVersion: v1 kind: Pod metadata: name: cuda-vector-add spec: restartPolicy: OnFailure containers: - name: cuda-vector-add image: "k8s.gcr.io/cuda-vector-add:v0.1" resources: limits: nvidia.com/gpu: 1 nodeSelector: accelerator: nvidia-tesla-k80

Create the pod:

kubectl create -f nvidia-docker2-gpu-pod.yml

Enter the container and run nvidia-smi to confirm the GPU is visible and only one GPU is allocated.

5. Summary

Using nvidia-docker 1.0 requires manually mounting GPU drivers as volumes, which is error‑prone. nvidia-docker 2.0 leverages Kubernetes device plugins, eliminating manual driver mounting and simplifying GPU resource management. Kubernetes also exposes a generic device‑plugin interface for other hardware resources, showcasing its extensibility.