How to Dynamically Provision Ceph RBD Storage in Kubernetes with StorageClass

1. Deploy the external rbd‑provisioner client

Kubernetes clusters initialized with kubeadm lack the built‑in RBD commands, so an external provisioner is required. Create a YAML manifest ( rbd-provisioner.yaml) that defines the necessary RBAC objects and a Deployment using the image quay.io/external_storage/rbd-provisioner:latest:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
rules:
- apiGroups: [""]
  resources: ["persistentvolumes"]
  verbs: ["get","list","watch","create","delete"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get","list","watch","update"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create","update","patch"]
- apiGroups: [""]
  resources: ["services"]
  resourceNames: ["kube-dns","coredns"]
  verbs: ["list","get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
subjects:
- kind: ServiceAccount
  name: rbd-provisioner
  namespace: default
roleRef:
  kind: ClusterRole
  name: rbd-provisioner
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: rbd-provisioner
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get","list","watch","create","update","patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rbd-provisioner
subjects:
- kind: ServiceAccount
  name: rbd-provisioner
  namespace: default
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: rbd-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: rbd-provisioner
    spec:
      containers:
      - name: rbd-provisioner
        image: quay.io/external_storage/rbd-provisioner:latest
        env:
        - name: PROVISIONER_NAME
          value: ceph.com/rbd
      serviceAccount: rbd-provisioner
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rbd-provisioner

Apply the manifest:

kubectl apply -f rbd-provisioner.yaml

2. Create the Ceph secret for Kubernetes

On a Ceph monitor node, retrieve the client.admin keyring and encode it with base64: ceph auth get-key client.admin | base64 The output (example):

QVFBczlGOWRCVTkrSXhBQThLa1k4VERQQjhVT29wd0FnZkNDQmc9PQ==

Store this value in a Kubernetes Secret named ceph-secret:

apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
type: "ceph.com/rbd"
data:
  key: QVFBczlGOWRCVTkrSXhBQThLa1k4VERQQjhVT29wd0FnZkNDQmc9PQ==

Apply the secret:

kubectl apply -f ceph-secret.yaml

3. Define a StorageClass that uses the Ceph RBD provisioner

First, create a pool in the Ceph cluster (executed on a Ceph monitor node):

ceph osd pool create k8stest 256
rbd create rbda -s 1024 -p k8stest
rbd feature disable k8stest/rbda object-map fast-diff deep-flatten

Then create storageclass.yaml with the following content:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: k8s-rbd
provisioner: ceph.com/rbd
parameters:
  monitors: 192.168.199.201:6789
  adminId: admin
  adminSecretName: ceph-secret
  pool: k8stest
  userId: admin
  userSecretName: ceph-secret
  fsType: xfs
  imageFormat: "2"
  imageFeatures: "layering"

Key parameter explanations:

monitors : comma‑separated list of Ceph monitor addresses.

pool : the Ceph pool created earlier.

userId / userSecretName : credentials for Kubernetes to access Ceph.

fsType : filesystem type for the RBD block device (e.g., xfs or ext4).

imageFormat and imageFeatures : usually left at defaults.

Apply the StorageClass:

kubectl apply -f storageclass.yaml

4. Deploy a StatefulSet that consumes the dynamic storage

Create state.yaml defining a headless Service and a StatefulSet that references the k8s-rbd StorageClass:

apiVersion: v1
kind: Service
metadata:
  name: storage
  labels:
    app: storage
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: storage
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: storage
spec:
  serviceName: "storage"
  replicas: 2
  template:
    metadata:
      labels:
        app: storage
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      volumeMode: Filesystem
      storageClassName: k8s-rbd
      resources:
        requests:
          storage: 1Gi

Apply the StatefulSet: kubectl apply -f state.yaml After execution, Kubernetes automatically creates a PersistentVolumeClaim (PVC) for each replica, and the associated PersistentVolumes (PV) are provisioned dynamically by the Ceph RBD provisioner based on the defined StorageClass.