Automate etcd Snapshots and Store to MinIO with Kubernetes CronJobs

Introduction

Two Kubernetes clusters are used: a Raspberry Pi running k3s with MySQL, and an x86 NUC running a standard Kubernetes. The NUC's etcd had never been backed up, so a backup process is created.

Procedure

The backup is performed with an etcd snapshot taken via the etcdctl CLI, then uploaded to MinIO.

Python script

Imports os, time, Minio and S3Error. Configuration parameters include the current date, etcd endpoint, certificate paths, and backup file name.

import os
import time
from minio import Minio
from minio.error import S3Error
# Get current time
now = time.strftime("%Y%m%d", time.localtime())
# Configuration parameters
etcd_url = ""
cacert = "./ssl/ca.pem"
cert = "./ssl/node-node1.pem"
key = "./ssl/node-node1-key.pem"
backup_file_name = "etcd-" + now

def create_snapshot():
    command = "ETCDCTL_API=3 etcdctl --endpoints=" + etcd_url + " --cacert=" + cacert + " --cert=" + cert + " --key=" + key + " snapshot save " + backup_file_name
    os.system(command=command)

def upload_to_minio():
    client = Minio("oss.example.cn:9000", access_key="", secret_key="", secure=False)
    if not client.bucket_exists("etcd-backup"):
        client.make_bucket("etcd-backup")
    client.fput_object("etcd-backup", backup_file_name, backup_file_name)

if __name__ == "__main__":
    try:
        create_snapshot()
    except Exception as e:
        print(e)
    try:
        upload_to_minio()
    except S3Error as e:
        print("upload failed: " + e)

The script is scheduled with a Kubernetes CronJob.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: cronjob
spec:
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - image: registry.bboysoul.cn/rpi/etcd-backup:...
            name: etcd-backup
          restartPolicy: Never
  schedule: "0 3 * * *"
  successfulJobsHistoryLimit: 5

A Dockerfile builds an ARM image containing ArgoCD and kubectl.

FROM debian:stable-slim
RUN apt update -y && apt install git -y && rm -rf /var/lib/apt/lists/*
COPY ./argocd /bin
COPY ./kubectl /bin

A Drone pipeline builds the image, pushes it to a private registry, updates the CronJob manifest, commits and pushes the changes, and syncs the ArgoCD application.

kind: pipeline
type: kubernetes
name: build
platform:
  os: linux
  arch: arm
steps:
- name: docker
  image: plugins/docker
  settings:
    repo: registry.bboysoul.cn/rpi/etcd-backup
    registry: registry.bboysoul.cn
    tags:
    - latest
    - ${DRONE_COMMIT_SHA}
# ... additional steps for git, kubectl, and argocd commands ...

An ArgoCD Application resource points to the Git repository containing the CronJob definition.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: 'etcd-backup'
spec:
  destination:
    namespace: 'cronjob'
    server: '...'
  source:
    path: './etcd-backup'
    repoURL: '...'
    targetRevision: HEAD
  project: 'default'

Future work includes adding notifications after backup completion.