How Kubernetes Implements Leader Election for High‑Availability Controllers

Background

Components such as kube-controller-manager, kube-scheduler, and the controller-runtime library support leader election in high‑availability systems. The article starts by describing the command‑line flags used by kube-controller-manager for leader election, including --leader-elect, --leader-elect-renew-deadline, and others.

--leader-elect                               Default: true
--leader-elect-renew-deadline duration       Default: 10s
--leader-elect-resource-lock string          Default: "leases"
--leader-elect-resource-name string          Default: "kube-controller-manager"
--leader-elect-resource-namespace string    Default: "kube-system"
--leader-elect-retry-period duration        Default: 2s

The Kubernetes API provides a simple leader election mechanism based on two object attributes: ResourceVersion and Annotations . Annotations such as control-plane.alpha.kubernetes.io/leader identify the current leader, but this approach adds pressure to the API server and etcd.

Election in controller‑runtime

The controller-runtime package implements leader election in pkg/leaderelection. The key configuration struct is:

type Options struct {
    LeaderElection               bool
    LeaderElectionResourceLock  string
    LeaderElectionNamespace     string
    LeaderElectionID           string
}

Election starts via leaderelection.RunOrDie, which creates a LeaderElector using a resourcelock implementation (e.g., LeaseLock). The example shows the entry point RunOrDie() and the callbacks OnStartedLeading, OnStoppedLeading, and OnNewLeader.

Resource lock abstraction

The interface defined in tools/leaderelection/resourcelock/interface.go provides methods Get, Create, Update, RecordEvent, Identity, and Describe. Four lock types are available:

LeaseLock

ConfigMapLock

MultiLock

EndpointLock

LeaseLock implementation

LeaseLock uses the Lease resource in the coordination.k8s.io/v1 API. Key methods:

func (ll *LeaseLock) Get(ctx context.Context) (*LeaderElectionRecord, []byte, error) {
    ll.lease, err = ll.Client.Leases(ll.LeaseMeta.Namespace).Get(ctx, ll.LeaseMeta.Name, metav1.GetOptions{})
    if err != nil { return nil, nil, err }
    record := LeaseSpecToLeaderElectionRecord(&ll.lease.Spec)
    recordByte, err := json.Marshal(*record)
    if err != nil { return nil, nil, err }
    return record, recordByte, nil
}

Similarly, Create creates a new lease, Update updates the lease spec, and RecordEvent logs election events.

Election workflow

The election loop is driven by LeaderElector.Run. It repeatedly calls tryAcquireOrRenew until it becomes the leader or the context is cancelled. The core logic builds a LeaderElectionRecord, attempts to Get the existing lock, creates it if missing, or updates it if the current instance is the leader. Lease renewal uses fields AcquireTime, RenewTime, and LeaseDurationSeconds. If another holder’s lease has not expired, the instance backs off.

Building a HA application with leader election

A minimal Go program demonstrates how to use leaderelection with a LeaseLock. The program builds a Kubernetes client, creates a LeaseLock, and runs leaderelection.RunOrDie with callbacks that print a message when the pod is the leader.

package main

import (
    "context"
    "flag"
    "fmt"
    "os"
    "os/signal"
    "syscall"
    "time"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    clientset "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/clientcmd"
    "k8s.io/client-go/tools/leaderelection"
    "k8s.io/client-go/tools/leaderelection/resourcelock"
    "k8s.io/klog/v2"
)

func buildConfig(kubeconfig string) (*rest.Config, error) {
    if kubeconfig != "" {
        return clientcmd.BuildConfigFromFlags("", kubeconfig)
    }
    return rest.InClusterConfig()
}

func main() {
    klog.InitFlags(nil)
    var kubeconfig, leaseLockName, leaseLockNamespace, id string
    flag.StringVar(&kubeconfig, "kubeconfig", "", "path to kubeconfig")
    flag.StringVar(&id, "id", "", "holder identity")
    flag.StringVar(&leaseLockName, "lease-lock-name", "", "lease lock name")
    flag.StringVar(&leaseLockNamespace, "lease-lock-namespace", "", "lease lock namespace")
    flag.Parse()

    if leaseLockName == "" || leaseLockNamespace == "" {
        klog.Fatal("lease lock name and namespace must be provided")
    }
    config, err := buildConfig(kubeconfig)
    if err != nil { klog.Fatal(err) }
    client := clientset.NewForConfigOrDie(config)

    run := func(ctx context.Context) {
        for {
            fmt.Println("I am leader, working...")
            time.Sleep(5 * time.Second)
        }
    }

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()
    ch := make(chan os.Signal, 1)
    signal.Notify(ch, os.Interrupt, syscall.SIGTERM)
    go func() { <-ch; klog.Info("shutdown"); cancel() }()

    lock := &resourcelock.LeaseLock{
        LeaseMeta: metav1.ObjectMeta{Name: leaseLockName, Namespace: leaseLockNamespace},
        Client:    client.CoordinationV1(),
        LockConfig: resourcelock.ResourceLockConfig{Identity: id},
    }

    leaderelection.RunOrDie(ctx, leaderelection.LeaderElectionConfig{
        Lock:            lock,
        ReleaseOnCancel: true,
        LeaseDuration:   60 * time.Second,
        RenewDeadline:   15 * time.Second,
        RetryPeriod:    5 * time.Second,
        Callbacks: leaderelection.LeaderCallbacks{
            OnStartedLeading: func(ctx context.Context) { run(ctx) },
            OnStoppedLeading: func() { klog.Infof("leader lost: %s", id); os.Exit(0) },
            OnNewLeader: func(identity string) { if identity != id { klog.Infof("new leader elected: %s", identity) } },
        },
    })
}

The article also provides a Dockerfile for building the binary, a Kubernetes manifest that creates a ServiceAccount, Role, RoleBinding, and a Deployment with three replicas, and commands to inspect the created Lease objects.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-leaderelection
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: leaderelection
rules:
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: leaderelection
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: leaderelection
subjects:
- kind: ServiceAccount
  name: sa-leaderelection
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: leaderelection
spec:
  replicas: 3
  selector:
    matchLabels:
      app: leaderelection
  template:
    metadata:
      labels:
        app: leaderelection
    spec:
      serviceAccountName: sa-leaderelection
      containers:
      - image: cylonchau/leaderelection:v0.0.2
        command: ["./elector"]
        args:
        - "-id=$(POD_NAME)"
        - "-lease-lock-name=test"
        - "-lease-lock-namespace=default"
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name

After deploying, the Lease resource shows the current holder (the pod name) and timestamps for acquisition and renewal, confirming that the leader continuously renews its lease while followers monitor it.

References

Parameters: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/

Simple leader election: https://kubernetes.io/blog/2016/01/simple-leader-election-with-kubernetes/

pkg/leaderelection: https://github.com/kubernetes-sigs/controller-runtime/tree/master/pkg/leaderelection

client-go/tools/leaderelection: https://github.com/kubernetes/client-go/tree/v0.24.0/tools/leaderelection

Example: https://github.com/kubernetes/client-go/blob/v0.24.0/examples/leader-election/main.go

Resource lock interface: https://www.cnblogs.com/Cylon/p/tools/leaderelection/resourcelock/interface.go

Lease API: https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/lease-v1/