How to Connect Kubernetes to Ceph for Dynamic PVC Provisioning

This blog introduces how to integrate a Ceph cluster with a Kubernetes cluster, using Ceph as the backend storage to enable dynamic PVC provisioning. The guide assumes a healthy Ceph cluster and a kernel version of 4.1.4 or higher.

Integration Steps

Create storage pool

ceph osd pool create mypool 128 128 # set PG and PGP to 128
ceph osd pool ls
mypool

Create secret objects

Create a Ceph user for Kubernetes and grant the necessary permissions.

ceph auth get-or-create client.kube mon 'allow r' osd 'allow class-read object_prefix rbd_children,allow rwx pool=mypool'

cat ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ceph-kube-secret
  namespace: default
data:
  key: "ceph auth get-key client.kube | base64"
type: kubernetes.io/rbd
---
apiVersion: v1
kind: Secret
metadata:
  name: ceph-admin-secret
  namespace: default
data:
  key: "ceph auth get-key client.admin | base64"
type: kubernetes.io/rbd

kubectl apply -f ceph-secret.yaml

Create StorageClass

cat ceph-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-storageclass
provisioner: kubernetes.io/rbd
parameters:
  monitors: 172.16.200.101:6789,172.16.200.102:6789,172.16.200.103:6789
  adminId: admin
  adminSecretName: ceph-admin-secret
  adminSecretNamespace: default
  pool: mypool
  userId: kube
  userSecretName: ceph-kube-secret
  userSecretNamespace: default
  fsType: ext4
  imageFormat: "2"
  imageFeatures: "layering"

kubectl apply -f ceph-storageclass.yaml
kubectl get sc
NAME                PROVISIONER    AGE
ceph-storageclass   kubernetes.io/rbd   87s

Create PVC

cat ceph-storageclass-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-test-claim
spec:
  storageClassName: ceph-storageclass
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

ReadWriteOnce: read‑write, mounted by a single node. ReadOnlyMany: read‑only, can be mounted by multiple nodes. ReadWriteMany: read‑write, can be mounted by multiple nodes.

Create test pod

cat ceph-busybox-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod
spec:
  containers:
  - name: ceph-busybox
    image: busybox:1.32.0
    command: ["/bin/sh","-c","tail -f /etc/resolv.conf"]
    volumeMounts:
    - name: ceph-volume
      mountPath: /usr/share/busybox
      readOnly: false
  volumes:
  - name: ceph-volume
    persistentVolumeClaim:
      claimName: ceph-test-claim

If the pod does not reach Running state, verify that the ceph-common package is installed on all nodes and that the storageclass provisioner includes the rbd command; the default kubernetes.io/rbd provisioner often lacks it.

Troubleshooting

The error “executable file not found in $PATH” appears because the default kube‑controller‑manager image from gcr.io does not contain the rbd subcommand. Deploy an external provisioner:

cat storageclass-fix-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rbd-provisioner
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rbd-provisioner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: rbd-provisioner
    spec:
      containers:
      - name: rbd-provisioner
        image: "quay.io/external_storage/rbd-provisioner:latest"
        env:
        - name: PROVISIONER_NAME
          value: ceph.com/rbd
      serviceAccountName: persistent-volume-binder

The ServiceAccount must be persistent-volume-binder ; the default default account lacks permission to list the required resources.

kubectl apply -f storageclass-fix-deployment.yaml

Update the existing StorageClass to use the external provisioner:

...
metadata:
  name: ceph-storageclass
#provisioner: kubernetes.io/rbd
provisioner: ceph.com/rbd
...

kubectl get sc
NAME                PROVISIONER    AGE
ceph-storageclass   ceph.com/rbd   64m

Recreate the PVC and pod:

kubectl delete -f ceph-storageclass-pvc.yaml && kubectl apply -f ceph-storageclass-pvc.yaml
kubectl delete -f ceph-busybox-pod.yaml && kubectl apply -f ceph-busybox-pod.yaml

After the pod reaches Running, verify the PV:

kubectl get pv
NAME ... STATUS Bound CLAIM default/ceph-test-claim STORAGECLASS ceph-storageclass ...

kubectl exec -it ceph-pod -- /bin/sh
# mount | grep share
/dev/rbd0 on /usr/share/busybox type ext4 (rw,...)

The corresponding RBD image appears in the mypool pool:

rbd ls -p mypool
kubernetes-dynamic-pvc-...

Common Errors

1. RBD image cannot be mapped as a block device:

Warning  FailedMount ... rbd: map failed exit status 110, rbd output: rbd: sysfs write failed
...

Cause: Linux kernel versions lower than 4.5 lack the required feature flag. Disable the flag with:

ceph osd crush tunables hammer