Step-by-Step Guide: Deploy Ceph‑CSI (v3.1.0) with RBD on Kubernetes

This article details how to deploy the ceph-csi driver (v3.1.0) in a Kubernetes cluster and use RBD as persistent storage.

Required environment (illustrated below):

Environment version information:

Kubernetes version:

$ kubectl get node
NAME       STATUS   ROLES   AGE   VERSION
sealos01   Ready    master  23d   v1.18.8
sealos02   Ready    master  23d   v1.18.8
sealos03   Ready    master  23d   v1.18.8

Ceph version:

$ ceph version
ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)

1. Create Ceph Pool

Create a new Ceph pool named kubernetes for Kubernetes to use:

$ ceph osd pool create kubernetes

pool 'kubernetes' created

List all pools:

$ ceph osd lspools

1 cephfs_data
2 cephfs_metadata
3 .rgw.root
4 default.rgw.control
5 default.rgw.meta
6 default.rgw.log
7 kubernetes

2. Create User

Create a dedicated Ceph user for Kubernetes and ceph‑csi:

$ ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'

[client.kubernetes]
    key = AQBnz11fclrxChAAf8TFw8ROzmr8ifftAHQbTw==

Retrieve the key later if needed:

$ ceph auth get client.kubernetes
exported keyring for client.kubernetes
[client.kubernetes]
  key = AQBnz11fclrxChAAf8TFw8ROzmr8ifftAHQbTw==
  caps mgr = "profile rbd pool=kubernetes"
  caps mon = "profile rbd"
  caps osd = "profile rbd pool=kubernetes"

3. Deploy ceph‑csi

Clone the latest release branch (v3.1.0):

$ git clone --depth 1 --branch v3.1.0 https://gitclone.com/github.com/ceph/ceph-csi

Modify the ConfigMap with the Ceph cluster ID (fsid) and monitor addresses (v1 protocol):

---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "154c3d17-a9af-4f52-b83e-0fddd5db6e1b",
        "monitors": [
          "172.16.1.21:6789",
          "172.15.1.22:6789",
          "172.16.1.23:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config

Create a dedicated namespace for the deployment: $ kubectl create ns ceph-csi Apply the ConfigMap:

$ kubectl -n ceph-csi apply -f csi-config-map.yaml

Create Secret

Generate a Secret using the user ID and cephx key:

cat <<EOF > csi-rbd-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi
stringData:
  userID: kubernetes
  userKey: AQBnz11fclrxChAAf8TFw8ROzmr8ifftAHQbTw==
EOF

Deploy the Secret:

$ kubectl apply -f csi-rbd-secret.yaml

RBAC Authorization

Replace the default namespace in all manifests with ceph-csi and create the required ServiceAccount, ClusterRole, and ClusterRoleBinding:

$ kubectl create -f csi-provisioner-rbac.yaml
$ kubectl create -f csi-nodeplugin-rbac.yaml

Create the PodSecurityPolicy objects:

$ kubectl create -f csi-provisioner-psp.yaml
$ kubectl create -f csi-nodeplugin-psp.yaml

Deploy CSI sidecar

Update the csi-rbdplugin-provisioner.yaml and csi-rbdplugin.yaml files with the appropriate KMS settings (shown in the images below) and create the sidecar deployment:

$ kubectl -n ceph-csi create -f csi-rbdplugin-provisioner.yaml

The sidecar pod includes six containers: external‑provisioner, external‑attacher, csi‑resizer, and the CSI RBD plugin.

Deploy RBD CSI driver

Finally, deploy the RBD CSI driver:

$ kubectl -n ceph-csi create -f csi-rbdplugin.yaml

The pod contains two containers: CSI node‑driver‑registrar and the CSI RBD driver.

Create StorageClass

cat <<EOF > storageclass.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: 154c3d17-a9af-4f52-b83e-0fddd5db6e1b
  pool: kubernetes
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi
  csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi
  csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard
EOF

The clusterID corresponds to the earlier fsid, and imageFeatures limits the image features to those supported by the kernel.

3. Try ceph‑csi

Kubernetes abstracts storage via PersistentVolume (PV) and PersistentVolumeClaim (PVC). The example creates a PVC, verifies the bound PV, and runs a demo pod:

$ kubectl apply -f pvc.yaml
$ kubectl get pvc
$ kubectl get pv
$ kubectl apply -f pod.yaml
$ kubectl exec -it csi-rbd-demo-pod -- bash
# inside the pod, write and read a file

List RBD images in the pool and inspect their features:

$ rbd ls -p kubernetes
csi-vol-d9d011f9-f669-11ea-a3fa-ee21730897e6

$ rbd info csi-vol-d9d011f9-f669-11ea-a3fa-ee21730897e6 -p kubernetes
... features: layering ...

Verify the block device mapping on the node and inside the container:

$ rbd showmapped
$ lsblk -l | grep rbd
$ kubectl exec -it csi-rbd-demo-pod -- lsblk -l | grep rbd

All steps complete successfully.

References:

[1] Latest release branch (v3.1.0): https://github.com/ceph/ceph-csi/tree/v3.1.0

[2] gitclone: https://gitclone.com