Automate Kubernetes Local Storage and Backup with Carina and Velero

Why do we still need local storage in the cloud‑native era? Although network‑based storage is often recommended for containerized workloads, its I/O performance and lack of dynamic mounting for certain middleware like RabbitMQ or Kafka make local disks preferable, especially when enhanced by Kubernetes automation.

Is there a more suitable backup‑restore solution for Kubernetes? Traditional backup methods either rely on server‑side snapshots or deploy dedicated agents, both of which struggle with the elasticity and pooling of containerized environments. A Kubernetes‑native solution should provide one‑click backup and fast recovery.

Overall Plan

Prepare a Kubernetes cluster with a master node (no workloads) and at least two worker nodes.

Deploy Carina, the cloud‑native local container storage solution, and test its automatic disk management.

Deploy Velero, the cloud‑native backup solution, and test data backup and restore capabilities.

Kubernetes Environment

Version: v1.19.14

Cluster size: 1 master, 2 workers

Disk mounting: only the root uses a separate disk initially.

Deploy Carina

1. Deployment script – refer to the official documentation; note API changes after version 1.22.

2. Prepare local raw disks: attach at least a 20 GiB raw disk to each worker (Carina reserves 10 GiB for metadata).

Apply SSD and HDD disks:

# gcloud apply ssd
# gcloud apply hdd

3. Verify Carina can discover the new disks by modifying the default scan policy ConfigMap:

# kubectl describe cm carina-csi-config -n kube-system
# ConfigMap data (excerpt)
{
  "diskSelector": ["sd+"],
  "diskScanInterval": "180",
  "diskGroupPolicy": "type",
  "schedulerStrategy": "spradout"
}

4. Confirm the local disks are recognized:

# kubectl get node u20-w1 -o template --template={{.status.capacity}}
# Output shows carina.storage.io/carina-vg-hdd:200 and carina-vg-ssd:10
# kubectl get node u20-w1 -o template --template={{.status.allocatable}}
# Shows allocatable capacities after accounting for metadata.

5. Carina automatically groups disks into SSD and HDD volume groups using lsblk and creates corresponding storage classes.

Test Carina PV Auto‑Allocation

1. Create a StorageClass to enable automatic PV creation:

# cat > storageclass.yaml <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-carina-sc
provisioner: carina.storage.io
parameters:
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
mountOptions:
- rw
EOF
kubectl apply -f storageclass.yaml

2. Deploy a simple Nginx application that uses a PVC backed by Carina:

# PVC definition (csi-carina-pvc-big, 10Gi)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-carina-pvc-big
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: csi-carina-sc
  volumeMode: Filesystem
---
# Deployment definition (mounts the PVC at /usr/share/nginx/html)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: carina-deployment-big
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-server-big
  template:
    metadata:
      labels:
        app: web-server-big
    spec:
      containers:
      - name: web-server
        image: nginx:latest
        volumeMounts:
        - name: mypvc-big
          mountPath: /usr/share/nginx/html
      volumes:
      - name: mypvc-big
        persistentVolumeClaim:
          claimName: csi-carina-pvc-big
          readOnly: false
EOF
kubectl apply -f pvc-and-deployment.yaml

3. Verify the PVC is bound and the node reports the correct allocatable space.

# kubectl get pvc
# kubectl get pv
# kubectl get node u20-w2 -o template --template={{.status.allocatable}}

4. Enter the Nginx container, create custom HTML files, and confirm they are served:

# kubectl exec -ti carina-deployment-big-xxxx -- /bin/bash
# cd /usr/share/nginx/html
# echo "hello carina" > index.html
# curl localhost
# echo "test carina" > test.html
# curl localhost/test.html

Deploy Velero

1. Download and install the Velero CLI (v1.7.0):

# wget https://github.com/vmware-tanzu/velero/releases/download/v1.7.0/velero-v1.7.0-linux-amd64.tar.gz
# tar -xzvf velero-v1.7.0-linux-amd64.tar.gz
# cp velero-v1.7.0-linux-amd64/velero /usr/local/bin/

2. Deploy MinIO as the object store backend (using Carina storage for the MinIO PVC):

# kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: minio
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio-storage-pvc
  namespace: minio
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
  storageClassName: csi-carina-sc
  volumeMode: Filesystem
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
  namespace: minio
spec:
  selector:
    matchLabels:
      component: minio
  template:
    metadata:
      labels:
        component: minio
    spec:
      containers:
      - name: minio
        image: minio/minio:latest
        args:
        - server
        - /storage
        - --config-dir=/config
        - --console-address ":9001"
        env:
        - name: MINIO_ACCESS_KEY
          value: "minio"
        - name: MINIO_SECRET_KEY
          value: "minio123"
        ports:
        - containerPort: 9000
        - containerPort: 9001
        volumeMounts:
        - name: storage
          mountPath: "/storage"
        - name: config
          mountPath: "/config"
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: minio-storage-pvc
      - name: config
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: minio
  namespace: minio
spec:
  type: NodePort
  ports:
  - name: console
    port: 9001
    targetPort: 9001
  - name: api
    port: 9000
    targetPort: 9000
    protocol: TCP
  selector:
    component: minio
EOF

3. Install Velero with the MinIO backend (using the service DNS name):

# velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.2.1 \
    --bucket velero \
    --secret-file ./minio-cred \
    --namespace velero \
    --use-restic \
    --use-volume-snapshots=false \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.minio.svc:9000

4. Verify Velero components and the Restic daemonset are running:

# kubectl get all -n velero

Use Velero to Backup Application and Data

Create a backup of the Nginx workload, including its PVC, using Restic for volume data:

# velero backup create nginx-backup \
    --selector app=web-server-big \
    --default-volumes-to-restic

Check backup status and logs (note potential DNS resolution issues when accessing MinIO):

# velero backup describe nginx-backup
# velero backup logs nginx-backup

The backup completes with a TTL of 720 hours (30 days) and one Restic snapshot.

Use Velero to Restore Application and Data

Delete the Nginx deployment and its PVC to simulate a failure.

Restore the backup:

# velero restore create --from-backup nginx-backup

Verify that the deployment, PVC, and PV are recreated and that the custom HTML files are still present inside the container.

# kubectl get po,svc,pvc,pv
# kubectl exec -ti <pod-name> -- /bin/bash -c "cat /usr/share/nginx/html/index.html"

The restored pod includes an init container restic-wait that handles volume restoration before the main container starts.

Reference Links

使用 Velero 备份还原 Kubernetes 集群

Official Carina documentation: https://github.com/carina-io/carina/blob/main/README_zh.md#%E5%BF%AB%E9%80%9F%E5%BC%80%E5%A7%8B

Velero supported providers: https://velero.io/docs/v1.7/supported-providers/

Velero Restic restore helper customization: https://velero.io/docs/v1.7/restic/#customize-restore-helper-container

Velero source code for Restic restore action: https://github.com/vmware-tanzu/velero/blob/main/pkg/restore/restic_restore_action.go