Distributed Leader Election and Kubernetes Lease

Distributed Leader Election

In distributed systems, services often run on multiple nodes to achieve high availability, but some data or tasks cannot be processed in parallel and therefore require a single designated node, called the leader, to perform those tasks or coordinate decisions. The mechanism that selects this leader among nodes is known as leader election.

Many well‑known projects adopt leader election, for example:

Etcd

Kafka

Elasticsearch

Zookeeper

Common algorithms include:

Paxos : a classic distributed consensus algorithm, often regarded as the foundational consensus protocol.

Raft : one of the most widely used consensus algorithms today; Etcd uses Raft, and newer versions of Elasticsearch and Kafka have also switched to Raft.

ZAB (Zookeeper Atomic Broadcast) : the consistency protocol used by Zookeeper, which also provides leader election.

Kubernetes Lease

In Kubernetes, core components such as kube-scheduler and kube-controller-manager need leader election to guarantee that only one scheduler makes scheduling decisions and only one controller manager processes resources at any given time.

Beyond core components, user applications may also require leader election. Kubernetes offers a special resource called Lease ("租约") to satisfy this generic need.

The diagram (omitted) shows that leader election in K8s is implemented by contending for a distributed lock represented by a Lease . The instance that acquires the lock becomes the leader and must continuously renew the lease to prove it is still alive; if the leader crashes, the lock is released and other candidates can compete for leadership.

The structure of a Lease object is simple:

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata: # object
spec:
  acquireTime: # time when the lease was acquired
  holderIdentity: # identity of the current holder
  leaseDurationSeconds: # duration the holder must wait before it can be forcibly taken over
  leaseTransitions: # number of times the lease has changed owners
  renewTime: # last time the lease was renewed by the holder

In principle, a Lease is no different from other Kubernetes resources; you could also use a ConfigMap or Endpoint as a distributed lock because Kubernetes performs compare‑and‑swap on the resourceVersion field. However, using a dedicated Lease is recommended.

Usage Example

The following Go example demonstrates how to use a Lease for leader election:

import (
    "context"
    "time"

    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/leaderelection"
    "k8s.io/client-go/tools/leaderelection/resourcelock"
)

func main() {
    config, err := rest.InClusterConfig()
    if err != nil {
        panic(err.Error())
    }

    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        panic(err.Error())
    }

    // Configure Lease parameters
    leaseLock := &resourcelock.LeaseLock{
        LeaseMeta: metav1.ObjectMeta{
            Name:      "my-lease",
            Namespace: "default",
        },
        Client: clientset.CoordinationV1(),
        LockConfig: resourcelock.ResourceLockConfig{
            Identity: "my-identity",
        },
    }

    // Configure Leader Election
    leaderElectionConfig := leaderelection.LeaderElectionConfig{
        Lock:          leaseLock,
        LeaseDuration: 15 * time.Second,
        RenewDeadline: 10 * time.Second,
        RetryPeriod:   2 * time.Second,
        Callbacks: leaderelection.LeaderCallbacks{
            OnStartedLeading: func(ctx context.Context) {
                // This instance is now the leader; execute leader‑only logic here
            },
            OnStoppedLeading: func() {
                // This instance lost leadership; perform cleanup if needed
            },
            OnNewLeader: func(identity string) {
                // A new leader has been elected
            },
        },
    }

    leaderElector, err := leaderelection.NewLeaderElector(leaderElectionConfig)
    if err != nil {
        panic(err.Error())
    }

    // Start Leader Election
    ctx := context.Background()
    leaderElector.Run(ctx)
}

References:

https://kubernetes.io/docs/concepts/architecture/leases/

https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/lease-v1/

https://pkg.go.dev/k8s.io/[email protected]/tools/leaderelection