Mastering the Kubernetes Control Plane: From Informer Source Code to a Production‑Ready Dynamic Gateway Operator

1. Why static gateway configs and simple watches break under high concurrency

During a flash‑sale, an e‑commerce platform observed that static gateway templates, manual traffic cuts, and HPA based only on CPU/Memory caused four problems: controller overload when events arrive fast, API server slowdown due to retries, mismatched data‑plane state, and unclear source of truth.

The root cause is not the YAML or CRD definitions but an insufficient understanding of how the Kubernetes control plane works.

2. Informer is an event pipeline built around shared state, not just callbacks

Many developers think of an Informer as a convenient way to register AddFunc, UpdateFunc, and DeleteFunc. In reality, an Informer maintains a local cache that mirrors the API server and feeds multiple controllers.

informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc:    onAdd,
    UpdateFunc: onUpdate,
    DeleteFunc: onDelete,
})

The pipeline consists of:

Reflector : performs an initial List and then a continuous Watch, handling connection loss and resource‑version expiration.

DeltaFIFO : queues object changes, not whole objects, so controllers see the semantics of state transitions.

Indexer : provides fast local look‑ups, decoupling controllers from the API server.

SharedProcessor : distributes events to multiple handlers without guaranteeing idempotence.

Key engineering facts:

The local cache is only an approximation of the authoritative state. Watch streams can be interrupted; resourceVersion expiration is normal.

3. Designing the state model for a dynamic gateway

Instead of starting with "what events to watch", start with "what state to maintain". The author proposes four orthogonal faces:

Declaration face : business intent such as routes, degradation targets, and thresholds.

Observation face : facts collected from the cluster (Service, EndpointSlice) and external systems (Kafka lag, Envoy ACK).

Execution face : actions like generating snapshots, pushing xDS, updating status, emitting events.

Governance face : safety mechanisms (retry back‑off, concurrency limits, circuit breakers, finalizers, audit).

A production‑grade DynamicGateway CRD should separate spec (desired state) from status (observed execution), and include fields such as LastAppliedHash and LastAckedVersion to verify that a push really took effect.

type DynamicGatewaySpec struct {
    Routes []RouteSpec `json:"routes"`
    Policy GatewayPolicy `json:"policy,omitempty"`
}

type RouteSpec struct {
    Name               string `json:"name"`
    Path               string `json:"path"`
    BackendService     string `json:"backendService"`
    DegradeService     string `json:"degradeService,omitempty"`
    KafkaConsumerGroup string `json:"kafkaConsumerGroup,omitempty"`
    KafkaLagThreshold  int64  `json:"kafkaLagThreshold,omitempty"`
    MaxQPS             int32  `json:"maxQps,omitempty"`
}

type GatewayPolicy struct {
    FullResyncInterval metav1.Duration `json:"fullResyncInterval,omitempty"`
    FailOpen           bool            `json:"failOpen,omitempty"`
    RequireEnvoyAck    bool            `json:"requireEnvoyAck,omitempty"`
}

type DynamicGatewayStatus struct {
    ObservedGeneration int64               `json:"observedGeneration,omitempty"`
    Phase              string              `json:"phase,omitempty"`
    Conditions         []metav1.Condition  `json:"conditions,omitempty"`
    KafkaLag           map[string]int64    `json:"kafkaLag,omitempty"`
    LastAppliedHash    string              `json:"lastAppliedHash,omitempty"`
    LastAckedVersion   string              `json:"lastAckedVersion,omitempty"`
    LastError          string              `json:"lastError,omitempty"`
}

Design rules: spec only describes the desired state; runtime decisions belong in status or separate snapshots.

Never modify spec.routes[].backendService directly for degradation; generate a new desired data‑plane config instead.

Record execution results (hashes, ACK versions) so the controller can tell whether the data‑plane actually applied the configuration.

4. Reconcile must be idempotent and multi‑step

The controller‑runtime library hides boilerplate but the reconciliation loop still follows a strict sequence:

Read the latest object and cached dependencies.

Handle deletion and finalizers.

Collect external observations (e.g., Kafka lag) into a read‑only ObservedState struct.

Build the desired snapshot from spec + cluster facts + external facts.

Compare the snapshot hash with LastAppliedHash; push to Envoy only on change.

Update status, emit events, record metrics.

Decide whether to requeue immediately, after a back‑off, or after a periodic resync.

func (r *DynamicGatewayReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var gw gatewayv1.DynamicGateway
    if err := r.Get(ctx, req.NamespacedName, &gw); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    if !gw.DeletionTimestamp.IsZero() {
        return r.reconcileDelete(ctx, &gw)
    }
    if controllerutil.AddFinalizer(&gw, gatewayFinalizer) {
        if err := r.Update(ctx, &gw); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{}, nil
    }
    observed, err := r.collectObservedState(ctx, &gw)
    if err != nil {
        return r.failAndRequeue(ctx, &gw, "CollectObservedStateFailed", err)
    }
    desired, err := r.buildDesiredSnapshot(ctx, &gw, observed)
    if err != nil {
        return r.failAndRequeue(ctx, &gw, "BuildDesiredSnapshotFailed", err)
    }
    if desired.Hash != gw.Status.LastAppliedHash {
        if err := r.xdsPusher.Push(ctx, desired); err != nil {
            return r.failAndRequeue(ctx, &gw, "PushSnapshotFailed", err)
        }
    }
    if err := r.updateStatusSuccess(ctx, &gw, observed, desired); err != nil {
        return ctrl.Result{}, err
    }
    return ctrl.Result{RequeueAfter: gw.Spec.Policy.FullResyncInterval.Duration}, nil
}

Helper functions illustrate the separation of concerns:

func decideBackend(route RouteSpec, backends []BackendEndpoint, lag int64, policy GatewayPolicy) ResolvedRoute {
    primary := healthy(backends, route.BackendService)
    degrade := healthy(backends, route.DegradeService)
    switch {
    case lag > route.KafkaLagThreshold && len(degrade) > 0:
        return newResolvedRoute(route, degrade, "degraded")
    case len(primary) > 0:
        return newResolvedRoute(route, primary, "primary")
    case policy.FailOpen:
        return newResolvedRoute(route, previousKnownGood(route), "fail-open")
    default:
        return rejectRoute(route, "no-healthy-backend")
    }
}

The author stresses that half‑successful paths (snapshot built but push fails, partial ACKs, status updated but events missing) must be explicitly handled; otherwise the controller appears reliable while the system diverges.

5. Governance and safety switches

To avoid runaway retries and state inconsistency, the controller should:

Use a unified key‑based work queue for Service, EndpointSlice, and the custom resource.

Set MaxConcurrentReconciles to limit parallelism.

Separate unrecoverable errors (set Condition=False) from retryable ones.

Cache external metric results with TTL to prevent hammering Kafka or Prometheus.

Provide explicit admin switches: read‑only mode, route freeze, degradation circuit breaker, and audit events for every change.

6. When an Operator is justified

The decision table summarises three scenarios:

Simple, low‑frequency routing can be managed with GitOps and native gateway configs.

CPU/Memory‑based scaling can rely on HPA/KEDA.

Complex coordination of service topology, business metrics, and data‑plane ACKs warrants a custom Operator.

7. Production checklist

Before shipping the Operator, verify the following items:

SharedInformer or controller‑runtime cache is used to avoid duplicate watches.

EndpointSlice is the primary source for backend addresses.

External samplers have timeout, cache TTL, and circuit‑breaker settings.

Spec and Status are cleanly separated.

Execution fields such as LastAppliedHash and AckedVersion are recorded.

Finalizer cleans up external side‑effects on deletion.

Non‑retryable and retryable errors are handled differently.

MaxConcurrentReconciles limits are set.

Admin knobs for read‑only, freeze, and degradation are exposed.

Failure paths (cache miss, ACK loss, metric fetch error) are exercised in tests.

8. Conclusion

Informer provides a shared‑state event pipeline; an Operator turns business rules into a declarative state machine that continuously reconciles until reality matches the desired state. By separating declaration, observation, execution, and governance, and by rigorously handling half‑success scenarios, a production‑grade dynamic gateway can be built on top of the Kubernetes control plane.