How to Persist Data in Kubernetes with Ceph RBD: Static and Dynamic PV Guide

Kubernetes Persistent Storage Overview

Kubernetes uses two API resources to manage durable storage: PersistentVolume (PV) and PersistentVolumeClaim (PVC) . A PV is a cluster‑wide storage object provisioned by an administrator; it defines capacity, storage type, and access mode and exists independently of Pods. A PVC is a user request for storage that binds to a matching PV, similar to how a Pod consumes a Node.

Access Methods

Direct access : volume details are exposed to the user; low portability and security.

Static PV : the administrator creates PV objects manually; PVCs must match size and access mode.

Dynamic PV : Kubernetes creates a PV on demand via a StorageClass when a PVC requests a specific class.

Static PV Setup with Ceph RBD

Prerequisites : a running Kubernetes cluster (v1.9.0) and a Ceph cluster (v10.2.10). Install the Ceph client on every node: # yum install -y ceph-common Create a Ceph secret that contains the base64‑encoded admin key:

apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
data:
  key: QVFBOFF2SlZheUJQRVJBQWgvS2cwT1laQUhPQno3akZwekxxdGc9PQ==

Apply it: # kubectl create -f ceph-secret.yaml Define a static PV that points to an existing RBD image:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: ceph-pv
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  rbd:
    monitors:
      - mon-hosts:6789
    pool: rbd
    image: ceph-image
    user: admin
    secretRef:
      name: ceph-secret
    fsType: ext4
    readOnly: false
  persistentVolumeReclaimPolicy: Recycle

Create it: # kubectl create -f ceph-pv.yaml Verify the PV status is Available (example screenshot):

Create a PVC that requests 2 GiB with ReadWriteOnce:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

Apply it: # kubectl create -f ceph-claim.yaml Check that the PVC becomes Bound (example screenshot):

Deploy a Pod that mounts the PVC :

apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod2
spec:
  containers:
    - name: ceph-busybox
      image: busybox
      command: ["sleep", "60000"]
      volumeMounts:
        - name: ceph-vol1
          mountPath: /usr/share/busybox
  volumes:
    - name: ceph-vol1
      persistentVolumeClaim:
        claimName: ceph-claim

Create the pod: # kubectl create -f ceph-pod.yaml When the pod reaches Running , data written to /usr/share/busybox persists after the pod is deleted and recreated.

Dynamic PV Setup with Ceph RBD

Create a dedicated Ceph pool for Kubernetes (e.g., kube) and a user with appropriate permissions:

# ceph osd pool create kube 1024
# ceph auth get-or-create client.kube mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=kube' -o ceph.client.kube.keyring

Create a secret for the kube user (reuse the admin secret if desired):

apiVersion: v1
kind: Secret
metadata:
  name: ceph-kube-secret
  namespace: default
data:
  key: QVFCbEV4OVpmaGJtQ0JBQW55d2Z0NHZtcS96cE42SW1JVUQvekE9PQ==
type: kubernetes.io/rbd

Apply it: # kubectl create -f ceph-kube-secret.yaml Define a StorageClass that uses the new pool:

kind: StorageClass
metadata:
  name: dynamic
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/rbd
parameters:
  monitors: 10.139.206.209:6789
  adminId: admin
  adminSecretName: ceph-secret
  adminSecretNamespace: kube-system
  pool: kube
  userId: kube
  userSecretName: ceph-kube-secret
  fsType: ext4
  imageFormat: "1"

Create it: # kubectl create -f rbd-storage-class.yaml StorageClass creation status (example screenshot):

Create a PVC that triggers dynamic provisioning (size 1 GiB, ReadWriteOnce):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Apply it: # kubectl create -f dynamic-ceph-pvc.yaml The PVC becomes Bound to a PV that the dynamic StorageClass created (screenshot):

Deploy a Pod using the dynamically provisioned PVC (same pod spec as the static case, referencing ceph-claim). # kubectl create -f dynamic-ceph-pod.yaml When the pod runs, data written to the mounted path persists across pod restarts.

Limitations and Access Modes

Ceph RBD supports the three standard PV access modes: ReadWriteOnce – single node read/write. ReadOnlyMany – multiple nodes read‑only. ReadWriteMany – multiple nodes read/write (not supported by RBD; requires CephFS).

Consequently, workloads that need true multi‑node read/write must use CephFS instead of RBD.

References

https://www.kubernetes.org.cn/3462.html

https://kubernetes.io/docs/concepts/storage/persistent-volumes/

https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/rbd/examples