Connecting Ceph Storage to Kubernetes for Dynamic PVC Provisioning

This article explains how to integrate a Ceph cluster with a Kubernetes cluster to use Ceph as the backend storage and achieve dynamic PVC provisioning.

Host List

master-1 (Ceph mon, osd) – IP 172.16.200.101 – Kernel 4.4.247-1.el7.elrepo.x86_64

master-2 (Ceph mon, osd) – IP 172.16.200.102 – Kernel 4.4.247-1.el7.elrepo.x86_64

master-3 (Ceph mon, osd) – IP 172.16.200.103 – Kernel 4.4.247-1.el7.elrepo.x86_64

node-1 (Ceph osd) – IP 172.16.200.104 – Kernel 4.4.247-1.el7.elrepo.x86_64

Prerequisites

Upgrade the kernel to version 4.1.4 or newer as recommended by Ceph.

root ~ >>> ceph health
HEALTH_OK

Ensure the Ceph cluster is healthy before proceeding.

Integration Process

Create a storage pool

root ~ >>> ceph osd pool create mypool 128 128 # set PG and PGP to 128
pool 'mypool' created

root ~ >>> ceph osd pool ls
mypool

Create secret objects

Create a Ceph user for Kubernetes and grant appropriate permissions.

root ~ >>> ceph auth get-or-create client.kube mon 'allow r' osd 'allow class-read object_prefix rbd_children,allow rwx pool=mypool'

Generate the secret YAML files.

apiVersion: v1
kind: Secret
metadata:
  name: ceph-kube-secret
  namespace: default
data:
  key: "ceph auth get-key client.kube | base64"
type: kubernetes.io/rbd
---
apiVersion: v1
kind: Secret
metadata:
  name: ceph-admin-secret
  namespace: default
data:
  key: "ceph auth get-key client.admin | base64"
type: kubernetes.io/rbd

Apply the secrets to the cluster.

root ~/k8s/ceph >>> kubectl apply -f ceph-secret.yaml

Create a StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-storageclass
provisioner: kubernetes.io/rbd
parameters:
  monitors: 172.16.200.101:6789,172.16.200.102:6789,172.16.200.103:6789
  adminId: admin
  adminSecretName: ceph-admin-secret
  adminSecretNamespace: default
  pool: mypool
  userId: kube
  userSecretName: ceph-kube-secret
  userSecretNamespace: default
  fsType: ext4
  imageFormat: "2"
  imageFeatures: "layering"

root ~/k8s/ceph >>> kubectl apply -f ceph-storageclass.yaml
root ~/k8s/ceph >>> kubectl get sc
NAME               PROVISIONER        AGE
ceph-storageclass  kubernetes.io/rbd  87s

Create a PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-test-claim
spec:
  storageClassName: ceph-storageclass
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

ReadWriteOnce: read‑write, mounted by a single node ReadOnlyMany: read‑only, can be mounted by multiple nodes ReadWriteMany: read‑write, can be mounted by multiple nodes

Create a test pod

apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod
spec:
  containers:
  - name: ceph-busybox
    image: busybox:1.32.0
    command: ["/bin/sh","-c","tail -f /etc/resolv.conf"]
    volumeMounts:
    - name: ceph-volume
      mountPath: /usr/share/busybox
      readOnly: false
  volumes:
  - name: ceph-volume
    persistentVolumeClaim:
      claimName: ceph-test-claim

Check the pod status; if it does not reach running, the default kubernetes.io/rbd provisioner may be missing the rbd binary.

rbd: create volume failed, err: failed to create rbd image: executable file not found in $PATH:

Troubleshooting

The error occurs because the gcr.io kube‑controller‑manager image does not contain the rbd command. Deploy an external provisioner to handle RBD volumes.

Deploy external provisioner

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rbd-provisioner
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rbd-provisioner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: rbd-provisioner
    spec:
      containers:
      - name: rbd-provisioner
        image: "quay.io/external_storage/rbd-provisioner:latest"
        env:
        - name: PROVISIONER_NAME
          value: ceph.com/rbd
        serviceAccountName: persistent-volume-binder

Note: the ServiceAccount must be persistent-volume-binder; the default account lacks permission to list resources.

root ~/k8s/ceph >>> kubectl apply -f storageclass-fix-deployment.yaml

Update the existing StorageClass

# modify provisioner
metadata:
  name: ceph-storageclass
provisioner: ceph.com/rbd

root ~/k8s/ceph >>> kubectl get sc
NAME               PROVISIONER    AGE
ceph-storageclass  ceph.com/rbd  64m

Recreate PVC and pod

root ~/k8s/ceph >>> kubectl delete -f ceph-storageclass-pvc.yaml && kubectl apply -f ceph-storageclass-pvc.yaml
root ~/k8s/ceph >>> kubectl delete -f ceph-busybox-pod.yaml && kubectl apply -f ceph-busybox-pod.yaml

The pod reaches Running state.

root ~/k8s/ceph >>> kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
ceph-pod   1/1     Running   0          47s

Verify PV

root ~/k8s/ceph >>> kubectl get pv
NAME                                          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS       REASON   AGE
persistentvolume/pvc-...                      5Gi        RWO            Delete            Bound    default/ceph-test-claim   ceph-storageclass          112s

The RBD image is mapped as a block device under /usr/share/busybox/.

root ~/k8s/ceph >>> kubectl exec -it ceph-pod -- /bin/sh
# mount | grep share
/dev/rbd0 on /usr/share/busybox type ext4 (rw,seclabel,relatime,stripe=1024,data=ordered)

List images in the mypool pool:

root ~/k8s/ceph >>> rbd ls -p mypool
kubernetes-dynamic-pvc-...

Common Errors

1. RBD image cannot be mapped as a block device.

Warning  FailedMount ... rbd: map failed: exit status 110, rbd output: rbd: sysfs write failed
...

Cause: Linux kernel < 4.5 lacks required feature flags. Disable the unsupported feature:

ceph osd crush tunables hammer

After applying the above steps, the Ceph storage is successfully integrated with Kubernetes, and dynamic PVC provisioning works as expected.