Mastering K8s StorageClass with Ceph: From Basics to Production‑Ready Deployment

K8s StorageClass Design and Ceph Integration: From Beginner to Production Deployment

Preface: In the cloud‑native era, storage is a key bottleneck for application performance. This article guides you through the design principles of K8s StorageClass and provides a hands‑on Ceph integration to boost cluster storage performance by up to 300%.

Why StorageClass Is the Soul of K8s Storage

Traditional operations often face these pain points:

Chaotic storage resource allocation: Manual PV creation is error‑prone and inefficient.

Multi‑tenant isolation difficulty: Different business lines cannot be effectively separated.

Expansion cumbersome: Scaling storage for growth requires extensive manual intervention.

StorageClass acts as the "intelligent scheduler" of K8s storage, solving these issues.

StorageClass Core Principle Analysis

# High‑performance SSD StorageClass configuration
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd-ssd
annotations:
  storageclass.kubernetes.io/is-default-class: "false"
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: b9127830-b4cc-4e86-9b1d-991b12c4b754
  pool: k8s-ssd-pool
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: ceph-csi-rbd-secret
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  csi.storage.k8s.io/controller-expand-secret-name: ceph-csi-rbd-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
  csi.storage.k8s.io/node-stage-secret-name: ceph-csi-rbd-secret
  csi.storage.k8s.io/node-stage-secret-namespace: kube-system
  reclaimPolicy: Delete
  allowVolumeExpansion: true
  volumeBindingMode: Immediate

🚀 Ceph Integration Practical: Zero‑Downtime Deployment Plan

Environment Preparation Checklist

Before starting, ensure the following conditions:

Kubernetes cluster: version 1.20+

Ceph cluster: Pacific 16.x or newer

Node specifications: each worker node at least 4 CPU / 8 GiB RAM

Network requirements: intra‑cluster bandwidth ≥1 Gbps

Step 1: Deploy Ceph‑CSI Driver

# Download official deployment files
curl -O https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
curl -O https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml

# Create dedicated namespace
kubectl create namespace ceph-csi-rbd

# Deploy CSI driver
kubectl apply -f csi-rbdplugin-provisioner.yaml
kubectl apply -f csi-rbdplugin.yaml

⚠️ Production Environment Notes:

Set resource limits for CSI Pods.

Enable pod anti‑affinity for high availability.

Configure monitoring and alerting mechanisms.

Step 2: Create Ceph Authentication Secret

# Create a dedicated user in the Ceph cluster
ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=k8s-pool'

# Retrieve the key and encode in base64
ceph auth get-key client.kubernetes | base64

# ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ceph-csi-rbd-secret
  namespace: kube-system
type: Opaque
data:
  userID: a3ViZXJuZXRlcw==   # base64 of "kubernetes"
  userKey: QVFBTmVsWmZ...   # your base64‑encoded key

Step 3: Configure ConfigMap

# ceph-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ceph-csi-config
  namespace: kube-system
data:
  config.json: |-
    [
      {
        "clusterID": "b9127830-b4cc-4e86-9b1d-991b12c4b754",
        "monitors": [
          "192.168.1.100:6789",
          "192.168.1.101:6789",
          "192.168.1.102:6789"
        ]
      }
    ]

💪 Advanced Feature Configuration

Multi‑Layer Storage Strategy Design

# High‑performance StorageClass – for databases
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd-high-perf
  labels:
    storage.tier: "high-performance"
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: b9127830-b4cc-4e86-9b1d-991b12c4b754
  pool: ssd-pool
  imageFeatures: layering,exclusive-lock,object-map,fast-diff
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
# Standard StorageClass – for general workloads
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd-standard
  labels:
    storage.tier: "standard"
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: b9127830-b4cc-4e86-9b1d-991b12c4b754
  pool: hdd-pool
  imageFeatures: layering
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate
---
# Cold StorageClass – for backup/archive
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd-cold
  labels:
    storage.tier: "cold"
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: b9127830-b4cc-4e86-9b1d-991b12c4b754
  pool: cold-pool
  imageFeatures: layering
reclaimPolicy: Retain
allowVolumeExpansion: false
volumeBindingMode: WaitForFirstConsumer

Smart PVC Template

# Database‑specific PVC template
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-data-pvc
  labels:
    app: mysql
    tier: database
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: ceph-rbd-high-perf
  resources:
    requests:
      storage: 100Gi
  selector:
    matchLabels:
      storage.tier: "high-performance"

🔧 Performance Tuning Secrets

Ceph Cluster Optimization

# 1. Adjust OSD thread pool size
ceph tell osd.* injectargs '--osd-op-threads=8'
ceph tell osd.* injectargs '--osd-disk-threads=4'

# 2. Optimize RBD cache
ceph tell osd.* injectargs '--rbd-cache=true'
ceph tell osd.* injectargs '--rbd-cache-size=268435456'   # 256 MB

# 3. Adjust PG count (important!)
# Formula: (OSD count × 100) / replica count / pool count
ceph osd pool set k8s-pool pg_num 256
ceph osd pool set k8s-pool pgp_num 256

K8s Node Optimization

# Optimize kernel parameters
cat >> /etc/sysctl.conf <<EOF
# RBD performance tuning
vm.dirty_ratio = 5
vm.dirty_background_ratio = 2
vm.swappiness = 1
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
EOF

sysctl -p

CSI Driver Resource Configuration

# csi-rbdplugin optimization
spec:
  containers:
  - name: csi-rbdplugin
    resources:
      limits:
        cpu: 1000m
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 128Mi
    env:
    - name: RBD_CACHE_ENABLED
      value: "true"
    - name: RBD_CACHE_SIZE
      value: "256Mi"

📊 Monitoring and Alert System

Prometheus Monitoring Configuration

# ceph-monitoring.yaml
apiVersion: v1
kind: ServiceMonitor
metadata:
  name: ceph-cluster-monitor
spec:
  selector:
    matchLabels:
      app: ceph-exporter
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics

Key Metric Alert Rules

# storage-alerts.yaml
groups:
- name: ceph.storage.rules
  rules:
  - alert: CephClusterWarningState
    expr: ceph_health_status == 1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Ceph cluster health warning"
      description: "Cluster {{ $labels.cluster }} is in Warning state"
  - alert: StorageClassProvisionFailed
    expr: increase(ceph_rbd_provision_failed_total[5m]) > 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Storage provision failure"
      description: "StorageClass {{ $labels.storage_class }} failed provisioning in the last 5 minutes"

🛡️ Production‑Grade Security Configuration

RBAC Permission Control

# ceph-csi-rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rbd-csi-provisioner
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: rbd-csi-provisioner-role
rules:
- apiGroups: [""]
  resources: ["persistentvolumes"]
  verbs: ["get","list","watch","create","delete"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get","list","watch","update"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses"]
  verbs: ["get","list","watch"]

Network Policy Isolation

# ceph-network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ceph-csi-network-policy
  namespace: kube-system
spec:
  podSelector:
    matchLabels:
      app: csi-rbdplugin
  policyTypes:
  - Ingress
  - Egress
  egress:
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: TCP
      port: 6789   # Ceph Monitor
    - protocol: TCP
      port: 6800   # Ceph OSD start
      endPort: 7300

🚨 Fault Troubleshooting Manual

Common Issue Diagnosis

Issue 1: PVC stuck in Pending

# Check if the StorageClass is correct
kubectl get storageclass

# Describe the PVC
kubectl describe pvc <pvc-name>

# Verify CSI driver pod status
kubectl get pods -n kube-system | grep csi-rbd

# View driver logs
kubectl logs -n kube-system <csi-rbdplugin-pod> -c csi-rbdplugin

Issue 2: Mount failure errors

# Test Ceph cluster connectivity
kubectl exec -it <csi-pod> -n kube-system -- rbd ls --pool=k8s-pool

# Verify secret configuration
kubectl get secret ceph-csi-rbd-secret -n kube-system -o yaml

# Check RBD module on node
lsmod | grep rbd
modprobe rbd   # load if missing

Performance Issue Diagnosis

# 1. Inspect Ceph cluster performance
ceph -s
ceph osd perf
ceph osd df

# 2. Benchmark storage
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: storage-benchmark
spec:
  containers:
  - name: benchmark
    image: nginx
    volumeMounts:
    - name: test-volume
      mountPath: /data
    command: ["/bin/sh","-c","dd if=/dev/zero of=/data/test bs=1M count=1024 oflag=direct && dd if=/data/test of=/dev/null bs=1M iflag=direct"]
  volumes:
  - name: test-volume
    persistentVolumeClaim:
      claimName: test-pvc
EOF

🎯 Best‑Practice Summary

Design Principles

Layered storage strategy: design different performance tiers based on business needs.

Capacity planning: reserve 20‑30% of cluster capacity for data rebalancing.

Backup strategy: use Ceph snapshots for application‑consistent backups.

Monitoring coverage: monitor cluster, StorageClass, and PV/PVC layers.

Operational Recommendations

Progressive rollout: validate in test environment, then deploy to production in batches.

Version management: keep Ceph‑CSI version compatible with Kubernetes version.

Documentation: record configuration parameters and change history.

Regular drills: conduct quarterly disaster‑recovery exercises.

Performance Benchmarks

IOPS: SSD pool can reach 30,000+ IOPS.

Bandwidth: up to 800 MB/s under network limits.

Latency: P99 latency < 10 ms.

🔮 Future Development Trends

CSI 2.0 specification: richer storage feature support.

Intelligent operations: AI‑driven storage resource scheduling.

Edge storage: distributed storage for edge‑computing scenarios.

Green computing: more energy‑efficient storage solutions.

💬 Closing Remarks

K8s StorageClass and Ceph integration is an essential skill for modern SRE and DevOps engineers. With this hands‑on guide, you should now master the full workflow from basic configuration to production‑grade optimization.