From Scripts to Operators: Building Reliable Go-based Kubernetes Auto-scaling Controllers

1. Scaling Latency: Find Where Time Is Spent

When traffic spikes, a Deployment's desired replicas may increase from 3 to 12, yet user timeouts persist. The end-to-end scaling latency can be decomposed into:

T_capacity = T_metric + T_decision + T_schedule + T_startup + T_ready

This decomposition aids diagnosis; stages may overlap. Key timestamps to record:

Desired replicas increased : Controller submitted scaling intent.

Pod created : Deployment/ReplicaSet controller began execution.

Pod Ready : Container passed readiness probe.

Business capacity restored : Instance serves traffic; caches, connection pools, JIT warmed up.

If latency concentrates in Java startup or node provisioning, rewriting the Operator won't eliminate those delays. Pre-warming, startup optimization, or proactive scaling may be more effective.

1.1 Don't Rewrite HPA Just for Custom Metrics

HPA (autoscaling/v2) supports custom and external metrics, multi-metric decisions, and scaling behavior configuration. Metric collection interval and controller sync period are distinct; a specific metrics-server configuration isn't universal.

CPU/Memory driven : Preferred solution: HPA. Custom controller extra cost: Reimplementing existing control logic.

QPS/Business metrics : Preferred solution: HPA + Metrics Adapter. Custom controller extra cost: Metric integration, fault tolerance, compatibility maintenance.

Queue or external events : Preferred solution: Evaluate KEDA. Custom controller extra cost: Triggers, activation, failure strategies.

Event warm-up, approval, budget, maintenance window constraints : Preferred solution: Business Operator, optionally generating HPA. Custom controller extra cost: CRD lifecycle, permissions, upgrades, on-call.

One-time bulk replica adjustment : Preferred solution: Script or ops task. Custom controller extra cost: Long-term controller may not pay off.

A custom Operator is justified when existing mechanisms cannot clearly express your continuous operational rules and your team can maintain those rules. This article uses a standalone experiment writing directly to the Deployment's scale subresource; production platforms may have the Operator manage HPA instead.

2. Define Who Owns Replica Count

If HPA, business Operator, GitOps, and manual scripts all modify replicas on the same Deployment, they create conflicting control loops. Leader election only coordinates controller instances sharing the same election domain; it doesn't prevent other writers.

2.1 Management Contract

Manage only apps/v1 Deployment in the same namespace as the AutoScaler.

Each Deployment binds to at most one enabled AutoScaler.

Before takeover, check existing HPA and other registered replica managers.

GitOps must relinquish spec.replicas management to avoid continuous overwrites.

Deleting AutoScaler stops management, retains Deployment and current replicas.

Initial version requires minReplicas >= 1; no scale-to-zero traffic activation.

Pause and dry-run are explicit policy states, not implemented by deleting Pods.

Concurrent binding checks need care: two CRs can pass a list-then-create check simultaneously. Production should implement a ScaleBinding resource with admission webhook rejecting existing HPA/conflicting bindings, and controller atomically patching ownerUID. Deleting CR unbinds only matching owner UID; target Deployment UID change pauses scale-down, revalidates ownership, and re-accumulates stabilization window. GitOps must still explicitly relinquish spec.replicas.

2.2 Don't Set ownerReference If You Don't Own the Workload

Deployments are usually created by the release system. AutoScaler merely references them; using Owns() would make them child objects, risking garbage collection of the workload when the CR is deleted. Owns(&Deployment{}) applies only to objects with matching ownerReference. A plain targetRef requires explicit mapping watches or periodic Reconcile.

3. Operator Is a Continuous Reconciliation Mechanism, Not an Event Callback Script

Reconcile receives object namespace/name, not ordered business events. The controller must re-read current state before deciding the next step. Idempotence means repeated processing under identical inputs, observations, and policy state produces no harmful side effects. Auto-scaling also depends on time and external metrics, so Reconcile isn't a pure function.

The pure function suitable for testing is the policy layer:

Metric samples + current replicas + policy config + historical state + current time → Decision

Informer/cache : Provides list/watch, local observation. Developer responsibility: watch scope, memory, stale data handling.

WorkQueue : Provides merge same key, retry scheduling. Developer responsibility: business rate limiting, external service concurrency budget.

Reconciler : Provides execute convergence logic. Developer responsibility: error classification, idempotence, side-effect recovery.

Manager : Provides lifecycle, leader election, endpoints. Developer responsibility: RBAC, probe semantics, exit behavior.

Manager's default client caches reads and writes directly, but not all reads go through cache; APIReader, subresources, and config change paths. A successful write followed by immediate cache read may not see the latest result.

4. CRD Is First a Business Contract

4.1 Example with Explicit Metric Semantics

apiVersion: autoscale.example.com/v1alpha1
kind: AutoScaler
metadata:
  name: order-service
  namespace: production
spec:
  targetRef:
    name: order-service
  minReplicas: 2
  maxReplicas: 20
  pollIntervalSeconds: 15
  paused: false
  dryRun: true
  metrics:
  - name: request-rate
    type: Prometheus
    metricProfileRef: order-service-total-qps
    targetPerReplica: "100"
    maxAgeSeconds: 45
    behavior:
      scaleUpMaxStep: 4
      scaleDownMaxStep: 1
      scaleDownStabilizationSeconds: 300
      scaleDownMinIntervalSeconds: 60

The contract specifies: query returns total QPS of the target service; targetPerReplica is the per-Ready-replica target throughput (requests/sec) validated by load testing; "100" is an experimental value, not a universal recommendation.

Prometheus address is centrally configured by admins; credentials via controlled Secrets. Ordinary CRs cannot specify arbitrary external URLs. metricProfileRef points to an admin-maintained MetricProfile that fixes the query expression, allowed labels/namespaces, timeout, max time range, and query cost budget. This prevents cross-tenant queries and high-cost queries from overloading shared Prometheus, and gives auditing/change review a concrete object.

4.2 API Design Validation

targetRef.name : Required; initial version fixed to same-namespace Deployment.

min/maxReplicas : 1 <= min <= max, with platform upper bound.

pollIntervalSeconds : Positive integer, with allowed range.

metrics : At least one, unique names, quantity upper bound.

metricProfileRef : Must exist, within authorization; Profile fixes query and cost bounds.

targetPerReplica : Parsable, finite, strictly > 0.

maxAgeSeconds : Positive; paired with query and collection pipeline.

behavior : Step sizes positive, time windows have upper bounds.

Using strings for thresholds, integer Minimum markers don't suffice; need Quantity type, CEL, or webhook, plus runtime parse validation. Cross-field relationships via CRD CEL, e.g.:

// +kubebuilder:validation:XValidation:rule="self.maxReplicas >= self.minReplicas",message="maxReplicas must be >= minReplicas"

Tooling (controller-gen) and cluster support must be confirmed in version baseline. targetRef existence still checked in Reconcile; admission-time existence doesn't guarantee it won't be deleted later.

4.3 Status Reflects Facts and Decisions, Not Confusion

status:
  observedGeneration: 7
  targetUID: "deployment-uid"
  targetReplicas: 6
  readyReplicas: 4
  recommendedReplicas: 10
  desiredReplicas: 10
  appliedReplicas: 6
  lastSuccessfulScaleTime: "2026-10-16T02:00:00Z"
  conditions:
  - type: MetricsAvailable
    status: "True"
    reason: SamplesValid
  - type: ScalingLimited
    status: "True"
    reason: ScaleUpStepLimited

Raw recommendation may exceed final desired; each field must reflect actual decision stage, not be collapsed to a single value for appearance. Conditions must include message, lastTransitionTime, and applicable observedGeneration. Suggest distinguishing at least: TargetAvailable, MetricsAvailable, AbleToScale, ScalingLimited. If Ready is provided, clarify whether it means "controller can manage" or "business replicas ready". status.observedGeneration == metadata.generation only indicates status evaluated against that spec version; it does not imply external metrics are unchanged. Cannot skip subsequent metric evaluation based on this.

5. Get the Capacity Formula Right

5.1 Total vs Average Metrics Cannot Be Mixed

Assume total service QPS = 900, per-replica target = 100 QPS: N_raw = ceil(900/100) = 9 Independent of current replica count (3 or 6). If input is per-Pod average, use: N_raw = ceil(N_current * M_average / M_target) Plugging sum(rate(...)) total QPS into the second formula and multiplying by current replicas double-counts capacity demand.

For backlog queues: if each consumer processes μ items/sec, want to clear backlog Q in T seconds with sustained arrival rate λ, start with: N_raw = ceil((λ + Q/T) / μ) Assumes scalable consumption, uniform task distribution, unsaturated downstream. Partition count, hot keys, DB connections may limit parallelism. Using ceil(Q / allowed_backlog_per_replica) is a backlog-target policy, not a clearance-time guarantee.

5.2 Metric Anomaly ≠ Zero Load

Prometheus integration must check: HTTP status, response size, JSON status, resultType, series count, value validity, freshness.

Empty vector: no valid observation, don't default to 0.

Multiple series: reject per contract or use predefined aggregation; don't arbitrarily pick first.

NaN, Inf, negative: illegal for QPS/queue semantics.

Timeout, permission error: record classification, don't enter zero-load branch.

Stale samples: reject for scale-down.

Instant query timestamp doesn't prove underlying scrape is fresh, especially with rate, aggregations, recording rules. Production should also check scrape target health, rule latency, or business heartbeat. maxAgeSeconds must complement these checks, not just compare query execution time.

5.3 Partial Metric Failure: Different Strategies for Scale-Up vs Scale-Down

Take max of valid metric recommendations; if any metric fails:

Valid metrics recommend scale-up: allow bounded scale-up.

Valid metrics recommend scale-down: keep current replicas, don't scale down.

All metrics fail: keep current replicas and alert.

Conservative strategy avoids monitoring failures triggering scale-down. Emergency floor replicas decided by business capacity contract; floor also subject to resource budget.

5.4 Testable Total-Quantity Decision Function

Standalone policy snippet in internal/scaling/decision.go; no Kubernetes client dependency:

package scaling

import (
	"fmt"
	"math"
)

type Metric struct {
	Value     float64
	PerReplica float64
	Valid     bool // ingestion layer already validated freshness and semantic correctness
}

type Decision struct {
	Replicas int32
	Degraded bool
}

func Recommend(current, min, max int32, metrics []Metric) (Decision, error) {
	if min < 1 || max < min || current < 0 || len(metrics) == 0 {
		return Decision{}, fmt.Errorf("invalid policy or input")
	}
	best := min
	valid, failed := 0, false
	for _, m := range metrics {
		if !m.Valid || math.IsNaN(m.Value) || math.IsInf(m.Value, 0) ||
			m.Value < 0 || math.IsNaN(m.PerReplica) ||
			math.IsInf(m.PerReplica, 0) || m.PerReplica <= 0 {
			failed = true
			continue
		}
		valid++
		raw := math.Ceil(m.Value / m.PerReplica)
		candidate := min
		// Clamp range before int32 conversion to avoid overflow from huge floats.
		if raw >= float64(max) {
			candidate = max
		} else if raw > float64(min) {
			candidate = int32(raw)
		}
		if candidate > best {
			best = candidate
		}
	}
	// On failure, prefer holding current value over shedding replicas, even if max was just lowered.
	// Platform should expose OutOfBounds, wait for metric recovery or manual approval before converging.
	if valid == 0 || (failed && best < current) {
		return Decision{Replicas: current, Degraded: true}, nil
	}
	return Decision{Replicas: best, Degraded: failed}, nil
}

This function only produces metric recommendations; stabilization windows, rate limiting, pause, and dry-run are handled upstream. Explicit layering avoids cramming all rules into one Reconcile.

6. Fast Scale-Up, Slow Scale-Down: Enforce Time Semantics

6.1 Stabilization Window, Step Size, and Cooldown Are Distinct Rules

Scale-up step : Problem solved: single change too large. Example: at most +4 this round.

Scale-up rate : Problem solved: high-frequency events accumulate scaling. Example: at most agreed amount in any 60s.

Scale-down stabilization window : Problem solved: short dips cause premature scale-down. Example: take max of recommended values over last 300s.

Scale-down minimum interval : Problem solved: continuous scale-down too fast. Example: at least 60s between successful scale-downs.

Tolerance : Problem solved: jitter near threshold. Example: can set bandwidth based on business measurement.

"At most +4 per round" ≠ "at most +4 per 15s": watch, retries may wake early. True time-rate limiting must use actual write records and time windows. Don't freeze all successful scale-ups for 60s; that prevents timely capacity addition under sustained load. Configure scale-up and scale-down separately.

6.2 Don't Lose Safety Memory on Restart

If scale-down stabilization window lives only in memory, new leader lacks history and may scale down immediately. Options:

Persist bounded, bucketed recommendation history to status or dedicated state object.

Don't save full history; on startup, target UID change, or key policy change, re-wait full window before allowing scale-down.

Option 2 is easier to get right but frequent restarts delay resource reclamation. Option 1 needs history length and write frequency limits; don't append to status on every event. Experimental v1 can use conservative option 2. Production must handle "scale write succeeded, status write failed" window; cannot rely on transactional status+scale.

7. Reconcile: Reliable Path from Read to Execute

Architectural pseudocode illustrating call order; evaluation, status update, and time-window methods need project implementation.

func Reconcile(ctx context.Context, key NamespacedName) (Result, error) {
    cr, err := readAutoScaler(ctx, key)
    if IsNotFound(err) { return Done(), nil }
    if err != nil { return Done(), err }
    if IsDeleting(cr) { return Done(), nil }

    if err := validatePolicy(cr.Spec); err != nil {
        return persistInvalidCondition(ctx, cr, err)
    }
    target, err := readTarget(ctx, cr)
    if err != nil { return classifyTargetFailure(ctx, cr, err) }
    if err := verifyManagementContract(ctx, cr, target); err != nil {
        return persistConflictCondition(ctx, cr, err)
    }
    if cr.Spec.Paused { return persistPaused(ctx, cr, target) }

    samples := fetchMetricsWithDeadline(ctx, cr)
    decision := evaluatePolicy(cr, target, samples, clock.Now())
    if !cr.Spec.DryRun && decision.NeedsWrite {
        if err := applyScaleWithPrecondition(ctx, cr, target, decision); err != nil {
            return Done(), err
        }
    }
    if err := patchStatusIfChanged(ctx, cr, target, decision); err != nil {
        return Done(), err
    }
    return After(nextEvaluationDelay(cr, decision)), nil
}

7.1 Classify Errors

CR deleted : Normal exit.

Invalid spec : Write InvalidSpec, wait for fix; no fast retry.

Deployment not found : Write TargetNotFound, retry per period or wait for mapping event.

403 Forbidden : Permission alert, retry per controlled period.

409 Conflict : Re-read, re-evaluate decision preconditions.

Prometheus failure : Degraded strategy: hold or scale-up, re-evaluate next cycle.

API temporarily unavailable : Return error, let queue backoff.

Kubernetes API permission errors return Forbidden; "silent failure" usually means app swallowed error, e.g. _ = r.Status().Update(...). Status write failures must be logged and propagated.

7.2 Narrow Write Surface via Scale Subresource

Use client-go typed client to access Deployment's /scale, avoiding full Deployment update:

// Snippet: kube is kubernetes.Interface, key is types.NamespacedName.
// expectedUID from this round's target Deployment read.
// expectedReplicas is desired replicas seen during policy evaluation.
s, err := kube.AppsV1().Deployments(key.Namespace).
    GetScale(ctx, key.Name, metav1.GetOptions{})
if err != nil { return err }
if s.UID != expectedUID || s.Spec.Replicas != expectedReplicas {
    return fmt.Errorf("target changed; evaluate again")
}
if s.Spec.Replicas != desired {
    s.Spec.Replicas = desired
    _, err = kube.AppsV1().Deployments(key.Namespace).
        UpdateScale(ctx, key.Name, s, metav1.UpdateOptions{})
    if err != nil { return err }
}

Retain GetScale's resourceVersion for API server conflict rejection. On conflict, re-enter evaluation; don't endlessly retry stale desired. Target UID detects same-name Deployment recreation; on UID change, reset or revalidate binding and stabilization window, don't reuse old workload's control history.

7.3 Write Status Only on Meaningful Changes

before := scaler.DeepCopy()
// Update fields; use meta.SetStatusCondition for standard Conditions.
if !apiequality.Semantic.DeepEqual(before.Status, scaler.Status) {
    err := r.Status().Patch(ctx, scaler,
        client.MergeFromWithOptions(before, client.MergeFromWithOptimisticLock{}))
    if err != nil { return ctrl.Result{}, err }
}

Imports from apimachinery api/equality and controller-runtime client. Don't modify LastTransitionTime every round; it marks condition transition time. Don't write high-frequency sampling times to status, turning every poll into an API write. Sampling freshness belongs in metric system; if business requires persistence, rate-limit it. Status subresource doesn't auto-prevent watch storms from status updates; avoid storms via diff writes, sensible predicates, and explicit polling paths.

7.4 Finalizer Based on Cleanup Responsibility

This example only stops management on CR deletion, no external resources to clean, so base design needs no Finalizer. If persistent target binding is introduced, unbinding becomes a designed cleanup responsibility, then consider Finalizer. Cleanup must be idempotent, completable even if target gone; need runbook for prolonged external system unavailability and permanent controller uninstall. Finalizer is a deletion barrier, not a mandatory template for every Operator.

8. Watching and Caching: Receive Changes, Control Cost

8.1 Reference Mapping Instead of Misusing Owns

Create field index for AutoScaler: index value = targetRef.name; namespace restricted by List's InNamespace. On Deployment change, find referencing CRs and enqueue.

// Illustrative: index registration and map func per project version.
ctrl.NewControllerManagedBy(mgr).
    For(&autoscalev1alpha1.AutoScaler{}, builder.WithPredicates(crPredicate)).
    Watches(&appsv1.Deployment{},
        handler.EnqueueRequestsFromMapFunc(mapDeploymentToScalers)).
    Complete(reconciler)

CR update predicate can watch generation, pause-related metadata, and deletion status. Pure GenerationChangedPredicate doesn't cover all metadata updates; if using Finalizer, also allow deletionTimestamp changes. External metrics produce no Kubernetes watch events, so each effective processing must schedule RequeueAfter. Periodic requeue cannot be skipped by observedGeneration check. Map function List errors must be logged and metered; periodic requeue as fallback, but don't hide mapping failures long-term.

8.2 Cache Limits Must Match Visibility Contract

Default cache scope may cover all watched resource types cluster-wide; new types may create informers. Large clusters must explicitly plan namespaces, object types, and selectors before rollout.

Watch only AutoScaler and necessary Deployments; don't cache all Pods for replica stats.

Use DefaultNamespaces to limit managed namespaces.

Enable ReaderFailOnMissingInformer when supported, to fail fast on unexpected reads.

Use APIReader only for few direct reads, controlling request frequency.

Don't cache unnecessary Secret contents; credential read path needs separate evaluation.

Label selectors trade off: if cache only includes labeled objects, unlabeled real objects appear as NotFound to cache client. Takeover must have explicit marking process; don't mistake this for target non-existence. ByObject with empty ConfigMap doesn't auto-become metadata-only. Metadata watch must use corresponding metadata objects and watch API; subsequent full-type reads may create full object cache.

9. Leader Election, Permissions, and Health Checks

9.1 Two Replicas Still Need Full Leader Election Permissions

Beyond CR and Deployment permissions, leader election needs access to Leases in controller namespace. Missing this, Pods run but cannot acquire leadership.

Split permissions:

Managed namespace : autoscalers — get/list/watch

Managed namespace : autoscalers/status — get/update/patch

Managed namespace : deployments — get/list/watch

Managed namespace : deployments/scale — get/update

Controller namespace : coordination.k8s.io/leases — get/list/watch/create/update/patch

Event namespace : core/events — create/patch

Whether CR main resource patch/finalizers permissions needed depends on actual updates. Controller doesn't create CRs, so no create/delete needed. Cross-namespace management can use ClusterRole + RoleBindings per namespace, or deploy namespace-isolated controllers. Cache limiting isn't permission isolation; both need simultaneous design.

9.2 Probes Answer Different Questions

/healthz

: Process alive; shouldn't restart due to transient Prometheus failure. /readyz: Instance completed required initialization; not equivalent to "is leader". healthz.Ping only returns process-level success; cannot prove cache sync, leader election, metric reads, or business control loop healthy. Initialization readiness maintained by corresponding lifecycle signals; business management capability observed via CR Conditions and monitoring. Standby instances not doing Reconcile can still be Ready. Don't let deployment flows waiting for all replicas Ready get stuck on "only leader Ready" definition.

10. Helm and CRD Upgrades Designed Separately

Suggested layout:

api/v1alpha1/          # API types, validation, generated code
internal/controller/   # Reconciliation and watch mapping
internal/scaling/      # Pure policy and tests
internal/metrics/      # Prometheus integration
config/crd/bases/      # CRD source artifacts
config/samples/        # Minimal workload and CR
charts/autoscale-operator/ # Deployment, RBAC, ServiceAccount

New projects can use Kubebuilder scaffolding:

mkdir autoscale-operator
cd autoscale-operator
kubebuilder init --domain example.com --repo example.com/platform/autoscale-operator
kubebuilder create api --group autoscale --version v1alpha1 --kind AutoScaler
make generate manifests

10.1 Record Version Combination

Retain Kubebuilder version, Go version, controller-runtime/client-go/apimachinery versions, controller-gen version, Kind node image digest. k8s.io/* dependencies must use compatible version sets; don't chase latest independently. Commit go.mod, go.sum, tool versions, and generated manifests. Article doesn't claim arbitrary versions compile; Manager Options, typed queue, metrics secure endpoints etc. must align with locked version docs.

10.2 CRD Doesn't Auto-Migrate with Helm

Helm crds/ CRDs aren't rendered as normal templates; existing CRD upgrades/deletions aren't handled by regular Helm lifecycle. More controlled flow:

CI checks CRD diff and existing CR compatibility.

Platform admin applies backward-compatible CRD changes.

Upgrade controller and validate old/new CRs.

If multi-version needed, design conversion webhook and storage version migration.

Rolling back binary doesn't auto-rollback CRD and stored objects. Deleting CRD affects corresponding custom resources; shouldn't be routine uninstall step. Deployment must configure non-root, security context, requests/limits, probes, graceful termination, and controlled metrics endpoint. Every Helm values must actually connect to template; no dead switches like prometheus.enabled.

11. Kind Acceptance: Make Experiments Falsifiable

Commands below are acceptance templates assuming project implementation; default controller in autoscale-operator-system, Deployment named autoscale-operator-controller-manager; adjust if scaffolding differs. Article hasn't executed these cluster experiments.

11.1 Verify Dependencies and Manifests

go test ./...
make generate manifests
helm lint ./charts/autoscale-operator
helm template autoscale-operator ./charts/autoscale-operator > /tmp/autoscale-rendered.yaml
kubectl apply --dry-run=server -f /tmp/autoscale-rendered.yaml

Server dry-run needs accessible cluster with CRDs pre-installed. Commands are checks, not proof that article includes Chart and all source files.

11.2 Build and Deploy

kind create cluster --name operator-dev
make docker-build IMG=autoscale-operator:v0.2.0
kind load docker-image autoscale-operator:v0.2.0 --name operator-dev
make install
make deploy IMG=autoscale-operator:v0.2.0
kubectl -n autoscale-operator-system rollout status deployment/autoscale-operator-controller-manager

Local images use IfNotPresent or Never to avoid remote pulls. Pick one deployment path; don't install two controllers via Kustomize and Helm simultaneously.

11.3 Establish First Closed Loop with Controllable Metric Source

Plain nginx doesn't provide http_requests_total. First test uses explicit test Prometheus fixture: serves /api/v1/query, strictly returns single vector, can switch values, empty, NaN, failure responses.

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {"metric": {}, "value": [1792116000, "900"]}
    ]
  }
}

Timestamp generated at runtime; above only shows format. Fixture for test only, injected via test config, not allowed as arbitrary URL for production CR.

Experiment conditions: min=2, max=20, per-replica target=100, initial replicas=2, dryRun=false. Stabilization window and step per policy enabled.

Total 900 : Raw recommendation 9, converge stepwise per scale-up step.

Total 10000 : Raw demand exceeds max, final capped at 20.

Total 0 : Wait scale-down window, then step down to 2 per interval.

Empty vector, NaN, timeout : Not interpreted as zero load, no false scale-down.

One fails, other recommends scale-up : Allow controlled scale-up, expose degraded status.

One fails, other recommends scale-down : Hold current replicas.

dryRun=true : Update recommendation, no scale write.

paused=true : No automatic replica adjustments.

Same-name target deleted and recreated : Detect new UID, revalidate management relationship.

Then integrate real Prometheus and load test, verifying scrape semantics, per-replica capacity, readiness latency, and business SLO.

11.4 Restart Experiments Must Distinguish Three Responsibility Layers

Delete business Pod : Recovery responsibility: Deployment/ReplicaSet. Acceptance focus: restore desired count; don't attribute to Operator.

Delete one controller Pod : Recovery responsibility: Deployment + leader election. Acceptance focus: management recovery, takeover latency, safety state restoration.

Metric backend failure : Recovery responsibility: scaling policy. Acceptance focus: freeze scale-down, alert, resume evaluation after recovery.

kubectl -n autoscale-operator-system get pods
# Pick a controller Pod from above, replace placeholder.
kubectl -n autoscale-operator-system delete pod <controller-pod-name>
kubectl -n production get deployment order-service -w

Before restart, create low load; check new leader honors scale-down protection. Only observing new Pod Running doesn't prove controller state recovered correctly.

11.5 Health Endpoints Must Hit Correct Port

kubectl -n autoscale-operator-system port-forward deployment/autoscale-operator-controller-manager 8081:8081

In another terminal:

curl --fail http://127.0.0.1:8081/healthz
curl --fail http://127.0.0.1:8081/readyz

Don't assume metrics Service exposes health ports; actual Service ports and targetPort are authoritative.

12. Testing Focuses on Failure Windows, Not Lines of Code

Policy unit tests must cover: total 900/100=9, max clamping, zero value, illegal thresholds, NaN/Inf, partial multi-metric failure, current replica out-of-bounds fault strategy.

Time-related tests inject Clock; no real sleep. Stabilization window, scale-down min interval, restart protection, policy changes need deterministic time sequences.

Controller integration tests should use envtest to verify status subresource, resourceVersion conflicts, field indexes, CRD validation. Fake client suits narrow tests but can't replace API server full behavior.

Especially verify two non-transactional windows:

Scale update succeeds, status update fails: next round must recover from facts, not re-amplify scale-up.

Metric evaluation succeeds, target changes before scale write: reject stale decision, re-evaluate.

Kind validates real control plane and Pod behavior; it doesn't represent production node provisioning, image registry performance, or Java app startup time.

13. Production Monitoring: Watch "Did Decision Take Effect?"

Reconcile latency, error rate, workqueue depth : Is controller backlogged or repeatedly failing?

Metric query latency, errors, scrape freshness : Is control basis trustworthy?

Recommended/desired/applied/ready : Stuck at recommendation, write, or business readiness?

Scale request conflicts and Forbidden : Multiple writers or missing permissions?

Times limited by bounds/stabilization : Policy too conservative or capacity insufficient?

Takeover latency, scale-down protection state : Does restart affect safety?

API request volume, memory, GC : State storms or cache bloat?

Log namespace/name, CR UID, generation, target UID, metric summary, raw recommendation, limited desired, actual write result, rejection reason. Don't log auth credentials; don't put full query expressions or error texts as high-cardinality metric labels.

Capacity planning estimate:

QPS_queries ≈ (managed_CR_count × metrics_per_CR) / polling_interval

Example: 1000 CRs, 2 metrics each, 15s interval → ~133 queries/sec, excluding retries and peaks. Leader switch may amplify instantaneous pressure via concentrated enqueue. Control with query timeout, limited concurrency, query deduplication or batching rules, jittered initial scheduling. MaxConcurrentReconciles increase only after calculating API server and Prometheus budgets.

14. Rollout Sequence: Prove Recommendations Before Granting Write Permission

Advance in phases, each with explicit exit criteria:

Offline replay : Feed historical load into policy, check recommended count, jitter, anomaly handling.

Shadow evaluation : Dry-run outputs recommendations, compare with existing scaling and business latency.

Limited takeover : Manage only explicitly handed-off low-risk services, cap max replicas and change rate.

Failure drills : Metric loss, API conflicts, leader switch, target recreation all meet expectations.

Expand scope : Retain pause entry, audit logs, manual takeover and recovery steps.

Manual takeover sequence must be clear: pause controller policy first, then adjust replicas, then decide when to resume auto-management. Otherwise manual changes get overwritten by next normal Reconcile.

The engineering value of auto-scaling isn't writing replicas from 3 to 12 once, but explaining every decision, limiting error impact, and recovering continuous management when load changes, metrics lie, APIs conflict, and controllers restart. Achieve that, and the Operator grows from a timed script into a system trusted with operational rules.

References and Verification Boundaries

Official references to cross-check during implementation. Web content changes with versions; production projects must also verify against locked dependencies and target Kubernetes version.

Kubernetes: Horizontal Pod Autoscaling — metric types, algorithm, behavior config.

Kubebuilder: Watching Resources — ownership watches and reference mapping.

controller-runtime: cache API and Client API — cache scope, missing informer, client read/write and Patch.

Kubebuilder: Using Finalizers — external cleanup and idempotent deletion.

Helm: Custom Resource Definitions — CRD install, upgrade, deletion boundaries.

Kubernetes: API Concepts — resourceVersion and concurrency control.

Prometheus: HTTP API — query response format and error semantics.