Mastering k0s: Deploy a Fully Automated HA Kubernetes Cluster with k0sctl

After two years of using kubeadm to deploy Kubernetes clusters, the author explores a more convenient binary deployment method with k0s , a simple, solid, and certified Kubernetes distribution.

1. k0s Overview

k0s is a downstream Kubernetes distribution that retains almost all native Kubernetes features, only omitting the in‑tree cloud provider. It compiles Kubernetes source code into binaries and runs them on the host, resulting in functionality identical to upstream Kubernetes.

2. Using k0sctl

k0sctl is a tool similar to kubeadm but with far greater extensibility. It connects to target hosts via SSH, uploads required files, and starts Kubernetes services to initialize the cluster.

2.1 Install a Cluster

First, install k0sctl:

# Install k0sctl
wget https://github.com/k0sproject/k0sctl/releases/download/v0.9.0/k0sctl-linux-x64
chmod +x k0sctl-linux-x64
mv k0sctl-linux-x64 /usr/local/bin/k0sctl

Then create a k0sctl.yaml configuration file:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - ssh:
      address: 10.0.0.11
      user: root
      port: 22
      keyPath: /Users/bleem/.ssh/id_rsa
    role: controller+worker
  - ssh:
      address: 10.0.0.12
      user: root
      port: 22
      keyPath: /Users/bleem/.ssh/id_rsa
    role: controller+worker
  - ssh:
      address: 10.0.0.13
      user: root
      port: 22
      keyPath: /Users/bleem/.ssh/id_rsa
    role: controller+worker
  - ssh:
      address: 10.0.0.14
      user: root
      port: 22
      keyPath: /Users/bleem/.ssh/id_rsa
    role: worker
  - ssh:
      address: 10.0.0.15
      user: root
      port: 22
      keyPath: /Users/bleem/.ssh/id_rsa
    role: worker
  k0s:
    version: 1.21.2+k0s.1
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: Cluster
      metadata:
        name: k0s
      spec:
        api:
          address: 10.0.0.11
          port: 6443
          k0sApiPort: 9443
          sans:
          - 10.0.0.11
          - 10.0.0.12
          - 10.0.0.13
        storage:
          type: etcd
        network:
          kubeProxy:
            disabled: false
            mode: ipvs

Run the apply command (ensure password‑less SSH access): k0sctl apply -c k0sctl.yaml After a short wait, a cluster with three masters and two workers is ready:

k1.node ➜ ~ k0s kubectl get node -o wide
NAME      STATUS   ROLES   AGE   VERSION          INTERNAL-IP   EXTERNAL-IP   OS-IMAGE               KERNEL-VERSION   CONTAINER-RUNTIME
k1.node   Ready    <none>   10m   v1.21.2+k0s      10.0.0.11     <none>        Ubuntu 20.04.2 LTS    5.4.0-77-generic   containerd://1.4.6
k2.node   Ready    <none>   10m   v1.21.2+k0s      10.0.0.12     <none>        Ubuntu 20.04.2 LTS    5.4.0-77-generic   containerd://1.4.6
k3.node   Ready    <none>   10m   v1.21.2+k0s      10.0.0.13     <none>        Ubuntu 20.04.2 LTS    5.4.0-77-generic   containerd://1.4.6
k4.node   Ready    <none>   10m   v1.21.2+k0s      10.0.0.14     <none>        Ubuntu 20.04.2 LTS    5.4.0-77-generic   containerd://1.4.6
k5.node   Ready    <none>   10m   v1.21.2+k0s      10.0.0.15     <none>        Ubuntu 20.04.2 LTS    5.4.0-77-generic   containerd://1.4.6

2.2 Extension Methods

k0sctl offers three extensibility mechanisms:

File upload – arbitrary files (binaries, images, scripts) can be uploaded before installation.

Manifests & Helm – files placed in /var/lib/k0s/manifests are applied as static pods; Helm charts can also be defined in the yaml.

Auxiliary scripts – hooks can run custom scripts on each host during installation.

2.3 Offline Image Packages

k0s can automatically import an offline image bundle placed in /var/lib/k0s/images/ into containerd:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - ssh:
      address: 10.0.0.11
      user: root
      port: 22
      keyPath: /Users/bleem/.ssh/id_rsa
    role: controller+worker
    files:
    - name: image-bundle
      src: /Users/bleem/tmp/bundle_file
      dstDir: /var/lib/k0s/images/
      perm: 0755

2.4 Switch CNI Plugin

k0s ships with Calico and kube‑router. To use Flannel, set the provider to custom and upload the Flannel yaml and CNI binaries:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - ssh:
      address: 10.0.0.11
      user: root
      port: 22
      keyPath: /Users/bleem/.ssh/id_rsa
    role: controller+worker
    files:
    - name: flannel
      src: /Users/bleem/tmp/kube-flannel.yaml
      dstDir: /var/lib/k0s/manifests/flannel
      perm: 0644
    - name: cni-plugins
      src: /Users/bleem/tmp/cni-plugins/*
      dstDir: /opt/cni/bin/
      perm: 0755
  k0s:
    version: v1.21.2+k0s.1
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: Cluster
      metadata:
        name: k0s
      spec:
        network:
          podCIDR: 10.244.0.0/16
          serviceCIDR: 10.96.0.0/12
          provider: custom

2.5 Upload k0s Binary

In offline environments, set uploadBinary: true and provide the binary path:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - ssh:
      address: 10.0.0.11
      user: root
      port: 22
      keyPath: /Users/bleem/.ssh/id_rsa
    role: controller+worker
    uploadBinary: true
    k0sBinaryPath: /Users/bleem/tmp/k0s

2.6 Change Image Versions

Component images can be overridden in the images section of the config:

k0s:
  version: v1.21.2+k0s.1
  config:
    images:
      kubeproxy:
        image: k8s.gcr.io/kube-proxy
        version: v1.21.3
      default_pull_policy: IfNotPresent

2.7 Adjust Master Component Parameters

Extra arguments for the API server, scheduler, and controller manager can be set via spec.api.extraArgs, spec.scheduler.extraArgs, and spec.controllerManager.extraArgs. Worker profiles can be customized through spec.workerProfiles.

3. k0s HA Setup

HA requires an external Layer‑4 load balancer that fronts the API server (6443), controller join API (9443), and Konnectivity ports (8132/8133). Example Nginx stream configuration:

error_log syslog:server=unix:/dev/log notice;
worker_processes auto;
stream {
  upstream kube_apiserver {
    least_conn;
    server 10.0.0.11:6443;
    server 10.0.0.12:6443;
    server 10.0.0.13:6443;
  }
  upstream konnectivity_agent {
    least_conn;
    server 10.0.0.11:8132;
    server 10.0.0.12:8132;
    server 10.0.0.13:8132;
  }
  upstream konnectivity_server {
    least_conn;
    server 10.0.0.11:8133;
    server 10.0.0.12:8133;
    server 10.0.0.13:8133;
  }
  upstream controller_join_api {
    least_conn;
    server 10.0.0.11:9443;
    server 10.0.0.12:9443;
    server 10.0.0.13:9443;
  }
  server { listen 0.0.0.0:6443; proxy_pass kube_apiserver; }
  server { listen 0.0.0.0:8132; proxy_pass konnectivity_agent; }
  server { listen 0.0.0.0:8133; proxy_pass konnectivity_server; }
  server { listen 0.0.0.0:9443; proxy_pass controller_join_api; }
}

3.2 Build HA Cluster

Full HA configuration (offline) example:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - ssh:
      address: 10.0.0.11
      user: root
      port: 22
      keyPath: /Users/bleem/.ssh/id_rsa
    role: controller+worker
    uploadBinary: true
    k0sBinaryPath: /Users/bleem/tmp/k0s
    files:
    - name: flannel
      src: /Users/bleem/tmp/kube-flannel.yaml
      dstDir: /var/lib/k0s/manifests/flannel
      perm: 0644
    - name: image-bundle
      src: /Users/bleem/tmp/bundle_file
      dstDir: /var/lib/k0s/images/
      perm: 0755
    - name: cni-plugins
      src: /Users/bleem/tmp/cni-plugins/*
      dstDir: /opt/cni/bin/
      perm: 0755
  # (repeat host entries for 10.0.0.12‑0.0.0.15)
  k0s:
    version: v1.21.2+k0s.1
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: Cluster
      metadata:
        name: k0s
      spec:
        api:
          externalAddress: 10.0.0.20
          sans:
          - 10.0.0.11
          - 10.0.0.12
          - 10.0.0.13
          - 10.0.0.20
        storage:
          type: etcd
        network:
          podCIDR: 10.244.0.0/16
          serviceCIDR: 10.96.0.0/12
          provider: custom
          kubeProxy:
            disabled: false
            mode: ipvs
        telemetry:
          enabled: false
        images:
          default_pull_policy: IfNotPresent

Execute k0sctl apply -c k0sctl.yaml and the HA cluster will be ready in a few minutes.

3.3 Certificate Renewal

k0s certificates are valid for one year; the CA defaults to ten years. In HA environments, simply restart k0scontroller.service before expiry to regenerate fresh certificates.

4. Backup and Restore

Run k0sctl backup to create a k0s_backup_TIMESTAMP.tar.gz file. Restore with k0sctl apply --restore-from k0s_backup_TIMESTAMP.tar.gz. The author notes that Velero may be a more reliable backup solution.

5. Other Advanced Features

5.1 etcd Replacement

For small clusters, k0s can replace etcd with kine backed by SQLite or MySQL:

apiVersion: k0s.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s
spec:
  storage:
    type: kine
    kine:
      dataSource: "sqlite:///var/lib/k0s/db/state.db?more=rwc&_journal=WAL&cache=shared"

5.2 Cluster User Management

Create a user with admin privileges:

k0s kubeconfig create --groups "system:masters" testUser > k0s.config

5.3 Containerd Configuration

Upload a custom containerd.toml to /etc/k0s/containerd.toml if special Containerd settings are required.

6. Conclusion

k0s provides a lightweight, binary‑based Kubernetes installation with almost no feature loss. Coupled with k0sctl’s extensibility, it offers a powerful, customizable deployment experience, though some details (e.g., disabling konnectivity) still need improvement.