Mastering k0sctl: Deploy a Fully Offline HA Kubernetes Cluster with k0s

1. Introduction to k0s

k0s is a downstream Kubernetes distribution that retains almost all upstream features, only omitting the in‑tree cloud provider.

2. Using k0sctl

k0sctl is a tool similar to kubeadm but with greater extensibility. It connects to target hosts via SSH, uploads files, and starts Kubernetes services.

2.1 Install k0sctl

# Install k0sctl
wget https://github.com/k0sproject/k0sctl/releases/download/v0.9.0/k0sctl-linux-x64
chmod +x k0sctl-linux-x64
mv k0sctl-linux-x64 /usr/local/bin/k0sctl

2.2 Create configuration file

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - ssh:
      address: 10.0.0.11
      user: root
      port: 22
      keyPath: /Users/bleem/.ssh/id_rsa
    role: controller+worker
  # ... other hosts ...
  k0s:
    version: 1.21.2+k0s.1
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: Cluster
      metadata:
        name: k0s
      spec:
        api:
          address: 10.0.0.11
          port: 6443
          k0sApiPort: 9443
          sans:
          - 10.0.0.11
          - 10.0.0.12
          - 10.0.0.13
        storage:
          type: etcd
        network:
          kubeProxy:
            disabled: false
            mode: ipvs

Alternatively generate a skeleton with k0sctl init --k0s > k0sctl.yaml and then apply with k0sctl apply.

3. Extending k0sctl

File upload: Upload binaries, offline image bundles, or custom scripts before installation.

Manifests & Helm: Files placed in /var/lib/k0s/manifests are applied as static pods; Helm charts can be defined in the yaml.

Hooks: Define hooks to run custom scripts on each host.

4. Offline HA deployment

k0s HA requires an external Layer‑4 load balancer that fronts ports 6443, 9443, 8132, and 8133. Example Nginx stream configuration is provided.

stream {
    upstream kube_apiserver {
        least_conn;
        server 10.0.0.11:6443;
        server 10.0.0.12:6443;
        server 10.0.0.13:6443;
    }
    # ... other upstreams ...
    server {
        listen 0.0.0.0:6443;
        proxy_pass kube_apiserver;
        proxy_timeout 10m;
        proxy_connect_timeout 1s;
    }
    # ... other servers ...
}

A full offline HA configuration yaml is shown, including externalAddress, etcd storage, custom CNI (Flannel), and telemetry disabled.

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - ssh:
      address: 10.0.0.11
      user: root
      port: 22
      keyPath: /Users/bleem/.ssh/id_rsa
    role: controller+worker
    uploadBinary: true
    k0sBinaryPath: /Users/bleem/tmp/k0s
    files:
    - name: flannel
      src: /Users/bleem/tmp/kube-flannel.yaml
      dstDir: /var/lib/k0s/manifests/flannel
      perm: 0644
    - name: image-bundle
      src: /Users/bleem/tmp/bundle_file
      dstDir: /var/lib/k0s/images/
      perm: 0755
    - name: cni-plugins
      src: /Users/bleem/tmp/cni-plugins/*
      dstDir: /opt/cni/bin/
      perm: 0755
  # ... other hosts ...
  k0s:
    version: v1.21.2+k0s.1
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: Cluster
      metadata:
        name: k0s
      spec:
        api:
          externalAddress: 10.0.0.20
          sans:
          - 10.0.0.11
          - 10.0.0.12
          - 10.0.0.13
          - 10.0.0.20
        storage:
          type: etcd
        network:
          podCIDR: 10.244.0.0/16
          serviceCIDR: 10.96.0.0/12
          provider: custom
          kubeProxy:
            disabled: false
            mode: ipvs
        telemetry:
          enabled: false
        images:
          default_pull_policy: IfNotPresent

Run k0sctl apply -c k0sctl.yaml to provision the cluster; the tool will upload files, download k0s binaries, and start services.

5. Certificate renewal

k0s uses a 10‑year CA; each restart regenerates one‑year certificates, so a simple systemctl restart k0scontroller.service refreshes them.

6. Backup and restore

Backup with k0sctl backup creates k0s_backup_TIMESTAMP.tar.gz. Restore with k0sctl apply --restore-from …, noting that restoration is equivalent to a fresh install.

7. Advanced features

Etcd replacement: Switch storage to SQLite or MySQL via the kine driver in the yaml.

User management: Create kubeconfig for a new user with

k0s kubeconfig create --groups "system:masters" testUser > k0s.config

.

Custom Containerd config: Upload a custom /etc/k0s/containerd.toml file via the files section.

Conclusion

k0s provides a near‑upstream Kubernetes experience with minimal feature loss, and k0sctl adds powerful, extensible automation for both online and fully offline installations, including HA, custom CNI, storage options, and backup capabilities.