Kubernetes HPA Scaling in Practice: From CPU Metrics to Custom Business Metrics

Introduction

HPA’s value lies in turning observable, capacity‑planned signals into stable replica decisions rather than merely adding pods when CPU is high. CPU utilization depends on resource requests; custom metrics rely on an adapter, label mapping, and metric semantics. Misconfiguration can cause no scaling, oscillation, quota exhaustion, or increased replicas without latency improvement.

Pre‑checks

1. Verify cluster version and HPA API

kubectl -n <namespace> version --client
kubectl -n <namespace> api-resources | grep -E '^horizontalpodautoscalers|metrics'

Use autoscaling/v2 when possible; actual API availability is determined by the cluster.

2. Ensure metrics-server is functional

kubectl -n kube-system get deployment metrics-server
kubectl -n <namespace> top pods
kubectl -n <namespace> top nodes

If top fails, CPU or memory HPA lacks a data source. Fix metrics-server, certificates, network, or resource collection before adjusting HPA.

3. Inspect workload requests and limits

kubectl -n <namespace> get deployment/<deployment-name> -o yaml
kubectl -n <namespace> describe deployment/<deployment-name>

CPU utilization percentage is calculated from requests; without a request, utilization‑based HPA cannot make correct decisions.

4. Review existing HPA and events

kubectl -n <namespace> get hpa
kubectl -n <namespace> describe hpa/<hpa-name>
kubectl -n <namespace> get events --sort-by=.lastTimestamp | tail -n 50

Fields AbleToScale, ScalingActive, ScalingLimited and related events are direct evidence of why scaling does or does not occur.

5. Check pod availability and PDB

kubectl -n <namespace> get pods -l app=<app-name>
kubectl -n <namespace> get pdb
kubectl -n <namespace> get deployment/<deployment-name> -o jsonpath='{.spec.replicas}
'

HPA only changes replica count; it cannot guarantee new pods are ready. Health checks, PDB and cluster capacity must be verified.

CPU‑based Scaling

6. Set container CPU requests

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <deployment-name>
  namespace: <namespace>
spec:
  template:
    spec:
      containers:
      - name: <container-name>
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "1"
            memory: "1Gi"

Requests should be based on stable load, cold‑start behavior, and capacity planning; setting them too low makes CPU utilization appear artificially high and triggers excessive scaling.

7. Dry‑run the Deployment

kubectl -n <namespace> apply --dry-run=server -f deployment.yaml
kubectl -n <namespace> diff -f deployment.yaml || true

Dry‑run does not write objects; a non‑empty diff only indicates differences. Still review impact before production rollout.

8. Create CPU‑based HPA (v2)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: <hpa-name>
  namespace: <namespace>
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: <deployment-name>
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

averageUtilization

is the average CPU usage relative to requests, not node CPU. minReplicas and maxReplicas must be chosen based on capacity, SLO and cost constraints.

9. Apply HPA and watch status

kubectl -n <namespace> apply -f hpa-cpu.yaml
kubectl -n <namespace> get hpa/<hpa-name> -w

Use watch for short‑term observation. After deployment, record business QPS, latency and error rate; do not rely solely on replica count.

10. Inspect HPA recommendation and conditions

kubectl -n <namespace> describe hpa/<hpa-name>
kubectl -n <namespace> get hpa/<hpa-name> -o yaml

If metrics are unknown, targets mismatch, or maxReplicas limits the recommendation, describe will show the reason and related events.

11. Examine pod CPU usage

kubectl -n <namespace> top pods -l app=<app-name> --containers
kubectl -n <namespace> get pods -l app=<app-name> -o wide

Usage must be evaluated together with requests, throttling and per‑pod request distribution; averages can hide hotspot pods.

Stability Tuning

12. Configure scaleUp and scaleDown behavior

behavior:
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
    - type: Percent
      value: 100
      periodSeconds: 60
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
    - type: Percent
      value: 25
      periodSeconds: 60
    selectPolicy: Max

Scale‑up can be aggressive; scale‑down should be conservative to avoid oscillation. Windows and percentages need validation against cold‑start, cache warm‑up and traffic cycles.

13. Full stable HPA example

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: <hpa-name>
  namespace: <namespace>
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: <deployment-name>
  minReplicas: 2
  maxReplicas: 10
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 25
        periodSeconds: 60
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

The complete manifest should be version‑controlled and rendered per environment. Avoid ad‑hoc edits in production.

14. Verify rollout interaction

kubectl -n <namespace> rollout status deployment/<deployment-name> --timeout=5m
kubectl -n <namespace> get rs -l app=<app-name>
kubectl -n <namespace> get pods -l app=<app-name>

During rollout, replicas change. Ensure maxSurge, maxUnavailable and HPA limits do not exceed cluster capacity.

15. Detect Pending pods caused by resource shortage

kubectl -n <namespace> get pods --field-selector=status.phase=Pending
kubectl -n <namespace> describe pod/<pending-pod-name>

If HPA wants more replicas but pods stay Pending, the root cause is usually node resources, taints, affinity or quota, not HPA parameters.

16. Check namespace resource quota

kubectl -n <namespace> get resourcequota
kubectl -n <namespace> describe resourcequota/<quota-name>

maxReplicas

must fit within quota and node capacity; otherwise autoscaling will continuously fail.

Custom Metrics

17. Verify custom‑metric API availability

kubectl -n <namespace> get --raw '/apis/custom.metrics.k8s.io/v1beta1' | jq .
kubectl -n <namespace> get --raw '/apis/external.metrics.k8s.io/v1beta1' | jq .

If the API is missing or lacks permission, first check the Prometheus adapter (or other adapter) deployment and RBAC. Do not fabricate metric names.

18. List metrics discovered by the adapter

kubectl -n <namespace> get --raw '/apis/custom.metrics.k8s.io/v1beta1/namespaces/<namespace>/pods/*/<metric-name>' | jq .

The metric name must match the name actually exposed by the adapter; raw Prometheus metric names and Kubernetes custom‑metric names may differ.

19. Configure Prometheus Adapter rules

rules:
  custom:
  - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace:
          resource: "namespace"
        pod:
          resource: "pod"
    name:
      matches: "^http_requests_total$"
      as: "http_requests_per_second"
    metricsQuery: 'sum(rate(http_requests_total{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

Adapter configuration applicability depends on exporter labels. Changing the adapter affects the whole cluster’s metric discovery; validate first in a test environment.

20. Apply adapter changes and roll restart

kubectl -n monitoring get configmap <adapter-config-name> -o yaml
kubectl -n monitoring rollout restart deployment/<adapter-deployment-name>
kubectl -n monitoring rollout status deployment/<adapter-deployment-name> --timeout=5m

Restarting the adapter briefly impacts custom‑metric queries. Save the config version before applying and expand the rollout only after verification.

21. Create HPA based on per‑pod QPS

metrics:
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target:
      type: AverageValue
      averageValue: "30"

Pod metrics are calculated as the average per pod and require the adapter to reliably associate the metric with the pod and namespace.

22. Create HPA based on external queue depth

metrics:
- type: External
  external:
    metric:
      name: <queue-depth-metric-name>
    target:
      type: AverageValue
      averageValue: "100"

External metrics suit queue depth or other out‑of‑cluster signals. Verify metric units, latency, partition semantics and deduplication.

23. Combine CPU and business metrics

metrics:
- type: Resource
  resource:
    name: cpu
    target:
      type: Utilization
      averageUtilization: 70
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target:
      type: AverageValue
      averageValue: "30"

When multiple metrics are present, HPA computes a replica recommendation for each and selects the larger value; each threshold must have capacity justification.

24. Cross‑validate raw metrics in Prometheus

# Use the actual exporter‑exposed metric
sum by (namespace, pod) (rate(http_requests_total{namespace="<namespace>"}[2m]))

PromQL validates that the original exporter provides data. It does not prove that the adapter has correctly mapped the metric; the custom.metrics API still needs checking.

Observation & Rollback

25. Observe replicas, latency and error rate

# Available replicas
sum(kube_deployment_status_replicas_available{namespace="<namespace>",deployment="<deployment-name>"})
# 95th‑percentile request latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{namespace="<namespace>"}[5m])) by (le))

If replica count rises while P95 latency also rises, the bottleneck is likely downstream (connection pool, rate‑limiting, cold start) and unlimited scaling should be avoided.

26. Audit an HPA change

kubectl -n <namespace> get hpa/<hpa-name> -o yaml > /tmp/<hpa-name>-after.yaml
kubectl -n <namespace> get events --sort-by=.lastTimestamp | tail -n 80

Saving the object and events helps determine when oscillation or metric loss began.

27. Pause automatic scaling

kubectl -n <namespace> patch hpa/<hpa-name> \
  --type merge -p '{"spec":{"minReplicas":<fixed-replicas>,"maxReplicas":<fixed-replicas>}}'

This online change must be preceded by confirming that the fixed replica count can handle current traffic. It is an emergency stop, not a long‑term configuration.

28. Restore original HPA manifest

kubectl -n <namespace> apply -f hpa-before.yaml
kubectl -n <namespace> describe hpa/<hpa-name>

Before restoring, ensure hpa-before.yaml comes from a controlled version and the target Deployment has not been renamed.

29. Delete a newly created HPA after confirmation

kubectl -n <namespace> delete hpa/<hpa-name> --dry-run=server -o yaml
kubectl -n <namespace> delete hpa/<hpa-name>

Deletion stops autoscaling immediately. Perform only after confirming another strategy is in place and manual replica count is safe; save the manifest before deletion.

30. Read current metrics and recommended replicas

kubectl -n <namespace> get hpa/<hpa-name> \
  -o jsonpath='{.status.currentMetrics}
{.status.desiredReplicas}
'

The output should be cross‑checked with describe hpa events, actual pod usage and business metrics. When current metrics are empty, manual replica settings must not mask a broken metric pipeline.

Acceptance, Risks & Rollback

Acceptance must cover metric availability, HPA conditions, pod scheduling, business latency, error rate, cluster capacity and replica change rate. CPU is only a starting point, not a universal signal; custom metrics should be used only after validating collection, labeling, units, windows and latency. Deleting or pausing HPA changes online elasticity; therefore fix a safe replica count, save the manifest, and execute within a traffic‑tolerant window.