How to Mount Ceph RBD and CephFS in Kubernetes: Step‑by‑Step Guide

Overview

Kubernetes can use Ceph as a storage backend either via block devices (RBD) or the Ceph file system (CephFS). The article presents two approaches for RBD: the traditional static PersistentVolume / PersistentVolumeClaim (PV & PVC) workflow and the newer dynamic provisioning using a StorageClass. It also shows a concise method to mount CephFS directly in a deployment.

Prerequisite

All Kubernetes nodes must have the ceph-common package installed (e.g., via yum install ceph-common ) so that the RBD mount command is available.

Static PV & PVC Method for Ceph RBD

Create the Ceph secret

# Get the admin key and base64‑encode it
ceph auth get-key client.admin | base64

Save the following as ceph-secret.yml (replace the key value with your base64‑encoded key):

apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
data:
  # base64‑encoded key
  key: QVFDaWtERlpzODcwQWhBQTdxMWRGODBWOFZxMWNGNnZtNmJHVGc9PQo=

Create the PersistentVolume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-pv
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  rbd:
    monitors:
      - 10.5.10.117:6789
      - 10.5.10.236:6789
      - 10.5.10.227:6789
    pool: data
    image: data
    user: admin
    secretRef:
      name: ceph-secret
    fsType: xfs
    readOnly: false
  persistentVolumeReclaimPolicy: Recycle

Create the PV:

kubectl create -f test.pv.yml

Create the PersistentVolumeClaim

kind: PersistentVolumeClaim
apiVersion: extensions/v1beta1
metadata:
  name: test-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

Create the PVC:

kubectl create -f test.pvc.yml

Deploy a pod that uses the PVC

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: test
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - name: test
        image: dk-reg.op.douyuyuba.com/op-base/openresty:1.9.15
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: "/data"
          name: data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: test-pvc

Apply the deployment:

kubectl create -f test.dm.yml

Dynamic Provisioning with StorageClass

Create a compatible secret

The secret type must be kubernetes.io/rbd for StorageClass usage:

# Example for the kube-system namespace
kubectl create secret generic ceph-secret \
  --type="kubernetes.io/rbd" \
  --from-literal=key='AQCikDFZs870AhAA7q1dF80V8Vq1cF6vm6bGTg=' \
  --namespace=kube-system

Create the StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: test-storageclass
provisioner: kubernetes.io/rbd
parameters:
  monitors: 192.168.1.11:6789,192.168.1.12:6789,192.168.1.13:6789
  adminId: admin
  adminSecretName: ceph-secret
  adminSecretNamespace: kube-system
  pool: data
  userId: admin
  userSecretName: ceph-secret

Create the StorageClass:

kubectl create -f test.sc.yml

Create a PVC that references the StorageClass

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-sc-pvc
  annotations:
    volume.beta.kubernetes.io/storage-class: test-storageclass
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

Create the PVC: kubectl create -f test.pvc.yml The deployment YAML is identical to the static‑PV example; it will automatically provision a PV via the StorageClass when the PVC is created.

Mounting CephFS Directly in a Deployment

The same secret created earlier can be reused. No separate PV or PVC is required; the CephFS volume is defined directly in the pod spec.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: test
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - name: test
        image: dk-reg.op.douyuyuba.com/op-base/openresty:1.9.15
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: "/data"
          name: data
      volumes:
      - name: data
        cephfs:
          monitors:
            - 10.5.10.117:6789
            - 10.5.10.236:6789
            - 10.5.10.227:6789
          path: /data
          user: admin
          secretRef:
            name: ceph-secret

Apply the deployment with:

kubectl create -f test.dm.yml

Key Points

Static PV & PVC requires pre‑creating a PersistentVolume that references Ceph monitors, pool, image, and secret.

Dynamic provisioning via StorageClass eliminates the need to create a PV manually; the PVC triggers PV creation.

When using StorageClass, the secret must be of type kubernetes.io/rbd.

CephFS can be mounted directly in a pod without a PV/PVC, simplifying the workflow.

All nodes must have ceph-common installed for RBD mounts to succeed.