How to Build a Highly Available Kubernetes Master on Private Cloud

HA Master Overall Architecture

We deployed a Kubernetes cluster with master high availability on a private‑cloud Ubuntu 14.04 platform. All control‑plane components except kubelet are run as static pods, and an external load balancer built from HAProxy and Keepalived provides a virtual IP for the masters.

Static Pods and kubelet

Static pods are managed directly by the kubelet on each node without the apiserver. The kubelet watches a /etc/kubernetes/manifests directory; adding, updating, or removing yaml files creates, updates, or deletes the corresponding static pods, ensuring their high availability.

kubelet service high availability

On Ubuntu 14.04 the kubelet is installed as an Upstart service. The Upstart configuration uses respawn and respawn limit to automatically restart the kubelet if it exits unexpectedly.

Core Technical Points and Challenges

Privileged mode for components

Components such as kubelet, apiserver, kube‑proxy, flannel, and keepalived need privileged access to kernel modules. The required privileges are enabled in the pod specifications.

Static pods must run on the host network

Control‑plane static pods use the host network to communicate via stable IP addresses and to modify routing rules, which is essential for HAProxy, kube‑proxy, and keepalived.

External Load Balancer deployment

The load‑balancer consists of at least two nodes running HAProxy and Keepalived in the same pod. Keepalived monitors HAProxy via a health‑check URL because container PID namespaces prevent direct signal checks.

Key steps include enabling the kernel IPVS module, setting net.ipv4.ip_forward=1, and allowing non‑local bind.

HAProxy configuration

HAProxy is configured for SSL termination (mode tcp) to support layer‑7 client affinity. The configuration files are mounted into the pod.

Keepalived configuration

Keepalived runs on each load‑balancer node with a keepalived.conf that defines the virtual IP and health‑check scripts.

Pitfalls Encountered in Practice

HAProxy Docker image wrapper

The official HAProxy image runs haproxy‑systemd‑wrapper, which adds the -Ds daemon option. In a container this causes the process to background and the container to exit. The workaround is to invoke HAProxy via its full path, e.g. /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg.

Container exit code 137

Exit code 137 indicates the container received a kill signal. Adjusting the HAProxy configuration (e.g., setting default‑dh‑param) resolved the warning and prevented the container from exiting.

Future Directions for HA Master

The community plans to support multi‑master configurations in upcoming Kubernetes releases (targeted for v1.4), which will allow native apiserver high availability without an external load balancer. Until then, the external load balancer approach remains valuable for private‑cloud scenarios, also enabling node‑port load balancing for worker nodes.

Q&A

Q1: How to mitigate kube‑proxy bottleneck?

A: Use NodePort on each node and place an external load balancer in front of the NodePorts to distribute traffic.

Q2: Does integrating LVS solve the load‑balancer issue?

A: Yes, containerizing the load balancer allows rapid deployment of a pod‑based LB for NodePort traffic.

Q3: Why does HAProxy use the systemd wrapper?

A: The wrapper adds the -Ds option to make HAProxy systemd‑compatible, preventing daemon forking.

References

https://github.com/kubernetes/kubernetes/issues/26852

https://github.com/kubernetes/kubernetes/pull/25428