Deploy Kubernetes Inside Kubernetes: Mastering Kubefarm and PXE Boot

Kubernetes-in-Kubernetes and Kubefarm Overview

This article is a translation of Andrei Kvapil's blog post, with some edits and reductions. It describes how, when managing two data centers, thousands of physical and virtual machines, and hundreds of thousands of sites, Kubernetes can declaratively describe both applications and infrastructure.

Working at the largest Czech hosting provider WEDOS Internet a.s, the author presents two projects: Kubernetes-in-Kubernetes and Kubefarm.

Infrastructure Architecture

Physical servers are split into two groups: control‑plane nodes and compute nodes. Control‑plane nodes run a stable OS and all cluster services, while compute nodes have no OS; they boot via PXE, download a rootfs image from the control plane, and then join the cluster with kubeadm join.

After the rootfs is downloaded, the node no longer needs a constant connection to the PXE server, allowing the PXE server to be safely restarted.

Kubefarm Project

Kubefarm aims to let anyone deploy infrastructure with Helm. It abandons the single‑cluster model because managing multiple development teams in one cluster is cumbersome and Kubernetes is not designed for multi‑tenant isolation.

Each team runs its own cluster, improving scalability and fault isolation while keeping the infrastructure fully declarative.

Kubefarm builds on Kubernetes-in-Kubernetes, uses LTSP as the PXE server, and automatically configures DHCP via dnsmasq-controller.

How It Works

The project follows the Twelve‑Factor App principles, treating Kubernetes itself as an application to be deployed inside Kubernetes.

Running Kubernetes in Kubernetes

Kubernetes-in-Kubernetes provides a ready‑made Helm chart that deploys a full Kubernetes control plane.

helm repo add kvaps https://kvaps.github.io/charts
helm install foo kvaps/kubernetes --version 0.13.1 \
  --namespace foo \
  --create-namespace \
  --set persistence.storageClassName=local-path

The chart’s value.yaml describes standard controller components (etcd, apiserver, controller‑manager, scheduler) and includes a management container with kubectl and kubeadm for cluster initialization and inspection.

Physical Server Orchestration

Compute nodes boot via PXE, receiving a pre‑built image that can be updated simply by rebuilding the image and rebooting the nodes.

Kubefarm’s value.yaml defines node pools, MAC addresses, IP addresses, and optional tags for DHCP options, Kubernetes labels, and taints.

Name: hostname

MAC‑address: per‑node MAC for PXE boot

IP‑address: for DHCP discovery

Example node pools include a generic pool of five nodes and a second pool with a single node tagged “debug” and “foo”.

Sample Practice

Updating the Helm chart in the examples/generic directory adds a PXE service and jobs that periodically generate new tokens for node registration.

ArgoCD visualizes the apiserver exposure; two separate LoadBalancer services are required for TFTP and HTTP.

Network Boot Image

The node image is built from a multi‑stage Dockerfile based on Ubuntu 20.04, installing kernel, LVM, systemd, SSH, and a container with kubelet and kubeadm. The final stage adds TFTP, Nginx, and GRUB to serve the boot image.

Webhooks and API Aggregation

Webhooks require the apiserver to reach the service directly. When the webhook runs in a separate cluster, Konnectivity can proxy traffic, allowing the apiserver to access external webhooks and kubelets.

Future Plans

The author hopes to move Kubernetes-in-Kubernetes into an official SIG Helm chart, integrate with the Machine Controller Manager for VM provisioning via KubeVirt, and explore Cluster‑API integration for automated physical Kubefarm clusters.