Effective Ways to Debug Kubernetes Components

Developing and debugging Kubernetes components is difficult because developers often work on the latest code, which requires matching versions of dependent components such as kubelet and kube-apiserver, and because component startup depends on other services and cluster configuration. Using an IDE for debugging also introduces service‑dependency problems.

Two Common Debugging Approaches

The author summarizes two frequently used methods: printing logs and redeploying the service to analyze execution logic, or attaching a debugger directly from an IDE.

Preparation Work

A Linux server (preferably Ubuntu 22.04) with at least 8 CPU cores, 16 GB RAM, and 150 GB disk.

Install the latest Go version, enable SSH, and allow root login.

Deploy a single‑node Kubernetes cluster using binary startup of core components to simplify debugging.

Installing a Kubernetes Cluster

The kubeasz tool can quickly create a single‑node cluster with binary‑started core components. The author prefers it over kubeadm because kubeadm launches core pods statically, which hinders debugging and requires image recompilation for log inspection.

$ export release=3.6.5
$ wget https://github.com/lengrongfu/kubeasz/releases/download/${release}/ezdown
$ chmod +x ./ezdown
$ ./ezdown -D
$ ./ezdown -S
$ docker exec -it kubeasz ezctl start-aio
$ source ~/.bashrc
$ kubectl version

Implementing the Requirement

Kubernetes primarily runs on Linux, so code that manipulates Linux‑specific logic cannot be compiled directly on macOS. The author presents two solutions: local development and remote development.

Local Development

Run a Linux container or an Ubuntu VM (e.g., via Multipass). Build an image that contains the development environment:

FROM ubuntu:22.04
RUN apt-get update
RUN apt-get install -y openssh-server
RUN sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
RUN sed -i 's/PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
RUN echo 'root:123456' | chpasswd
RUN apt-get install -y git vim curl wget cmake
RUN apt-get install -y software-properties-common
RUN add-apt-repository ppa:longsleep/golang-backports
RUN apt-get update
RUN apt-get install -y golang-go
RUN /etc/init.d/ssh start

Build and run the container, mounting the local Go workspace to preserve changes:

$ docker build -t golang-env:dev -f Dockerfile .
$ docker run -d -p 50024:22 -v /Users/{local}/goworspace/code/src:/root/go/src golang-env:dev /usr/sbin/sshd -D

This setup allows configuring Docker proxies and using plugins such as GitHub Copilot while developing locally.

Remote Development

Use a JetBrains IDE with the Remote Development feature:

Install any recent JetBrains IDE and open the Remote Development menu without a project.

Select SSH, provide username, host address, and port to create a project.

After connecting, add the project, choose the IDE version and project directory, then click “Start IDE and Connect”.

After the IDE connects, the Golang editing interface appears just as it would locally.

Testing the Requirement

The author uses Kubelet as an example. Since the core components installed by kubeasz are started with systemctl, the test can stop the service, compile the modified code, and restart the binary.

To test via log printing, copy the code to the server and run:

make all
# or compile a specific target
make all WHAT=cmd/kubelet GOFLAGS=-v

The resulting binary is usually found in _output/bin/, but the author observed it under ~/go/bin. Verify the version with:

~/go/bin/kubelet --version

Command‑Line Testing

Use ps to obtain the current kubelet start command and its parameters.

Stop the existing kubelet service: systemctl stop kubelet.service.

Start the compiled binary with the default parameters.

IDE Debugging

Again retrieve the start command via ps to capture parameters.

Stop the kubelet service with systemctl stop kubelet.service.

Enter the captured parameters into the IDE debugger and start a debug session.

References

Version skew policy: https://kubernetes.io/releases/version-skew-policy/

kubeasz quick start: https://github.com/lengrongfu/kubeasz/blob/master/docs/setup/quickStart.md