Auto‑Scaling LLM Inference with Kubernetes HPA Based on Request Queue Depth

Define Queue Metrics

The metric llm_queue_depth is a Gauge representing the number of requests waiting to enter the model execution stage; in‑flight requests are tracked separately by llm_inflight_requests. Cancelled, timed‑out, or error paths must decrement the counters, otherwise the HPA will be misled by permanently high values. The authoritative queue depth should be exposed by an external system such as Redis, Kafka, or a gateway, not estimated inside the application.

kubectl -n <namespace> version
kubectl -n <namespace> api-versions | grep '^autoscaling/'
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 -n <namespace> 2>/dev/null | jq '.resources | length'

FastAPI example shows how to register the two gauges and update them around the request handling logic. The wait_for_execution_slot and run_inference functions are placeholders for the real queue manager and inference engine.

import os
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
from prometheus_client import Gauge, start_http_server

namespace = os.environ["POD_NAMESPACE"]
pod_name = os.environ["POD_NAME"]
queue_depth = Gauge(
    "llm_queue_depth",
    "Requests waiting for model execution",
    ["namespace", "pod"],
)
inflight = Gauge(
    "llm_inflight_requests",
    "Requests currently executing",
    ["namespace", "pod"],
)

@asynccontextmanager
async def lifespan(app: FastAPI):
    start_http_server(9090)
    yield

app = FastAPI(lifespan=lifespan)

@app.post("/v1/generate")
async def generate(payload: dict):
    queue_depth.labels(namespace, pod_name).inc()
    try:
        await wait_for_execution_slot()
    except TimeoutError as exc:
        raise HTTPException(503, "queue timeout") from exc
    finally:
        queue_depth.labels(namespace, pod_name).dec()

    inflight.labels(namespace, pod_name).inc()
    try:
        return await run_inference(payload)
    finally:
        inflight.labels(namespace, pod_name).dec()

When multiple worker processes are used, a single‑process gauge cannot represent the pod‑wide total. Use prometheus_client multiprocess mode, a unified queue manager, or let the external queue emit the metric directly, and choose sum or max aggregation according to the implementation.

Deploy Application and Metrics Port

The Deployment injects pod identity via the Downward API and defines a startupProbe to avoid killing the pod during model loading.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
  namespace: <namespace>
spec:
  replicas: 2
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      containers:
      - name: server
        image: <image‑address>
        ports:
        - {name: http, containerPort: 8000}
        - {name: metrics, containerPort: 9090}
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef: {fieldPath: metadata.name}
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef: {fieldPath: metadata.namespace}
        resources:
          requests:
            cpu: "4"
            memory: 32Gi
            nvidia.com/gpu: "1"
          limits:
            cpu: "8"
            memory: 48Gi
            nvidia.com/gpu: "1"
        startupProbe:
          httpGet: {path: /health, port: http}
          periodSeconds: 10
          failureThreshold: 60
        readinessProbe:
          httpGet: {path: /health, port: http}
          periodSeconds: 5

The Service exposes both the application and the metrics endpoint.

apiVersion: v1
kind: Service
metadata:
  name: llm-inference
  namespace: <namespace>
  labels:
    app: llm-inference
spec:
  selector:
    app: llm-inference
  ports:
  - name: http
    port: 8000
    targetPort: http
  - name: metrics
    port: 9090
    targetPort: metrics

When using the Prometheus Operator, a ServiceMonitor is required.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: llm-inference
  namespace: <namespace>
spec:
  selector:
    matchLabels:
      app: llm-inference
  namespaceSelector:
    matchNames: [<namespace>]
  endpoints:
  - port: metrics
    path: /metrics
    interval: 15s
    scrapeTimeout: 10s

Validate from Application to Prometheus

First verify that the metrics are reachable inside the cluster, bypassing Prometheus and the Adapter.

kubectl run metrics-check -n <namespace> --rm -i --restart=Never \
  --image=curlimages/curl -- \
  curl -fsS http://llm-inference.<namespace>.svc:9090/metrics \
  | grep -E '^llm_(queue_depth|inflight_requests)'

The command creates a temporary pod, queries the /metrics endpoint, and filters the two custom metrics. Each metric must include correct namespace and pod labels.

In Prometheus, confirm that each target pod has a single time series:

llm_queue_depth{namespace="<namespace>"}
count by (namespace, pod) (
  llm_queue_depth{namespace="<namespace>"}
)

If the series is missing or duplicated, adjust the exporter or the aggregation method.

Configure Prometheus Adapter

The following values.yaml fragment works for the prometheus-community/prometheus-adapter chart. Verify the chart version with helm show values before applying.

prometheus:
  url: http://<prometheus‑service>.<prometheus‑namespace>.svc
  port: 9090
rules:
  default: false
  custom:
  - seriesQuery: 'llm_queue_depth{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace: {resource: namespace}
        pod: {resource: pod}
    name:
      matches: '^llm_queue_depth$'
    as: 'llm_queue_depth'
    metricsQuery: 'max by (<<.GroupBy>>) (llm_queue_depth{<<.LabelMatchers>>})'

The max aggregation avoids double‑counting when the same pod is scraped multiple times; however, duplicate scraping should still be fixed. Whether to use sum for multi‑worker metrics depends on how the gauge is maintained.

Because the Adapter is a shared component, upgrade it with a backup of the current values and a dry‑run.

helm get values prometheus-adapter -n <adapter‑namespace> \
  > prometheus-adapter.values.before.yaml
helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter \
  -n <adapter‑namespace> --create-namespace \
  -f adapter-values.yaml --dry-run \
  > /tmp/prometheus-adapter.rendered.yaml

Create Queue‑Based HPA

The example sets the target average queue depth per pod to 4. This value is not universal; it must be tuned according to request latency, concurrency slots, batch size, and cold‑start time.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-inference
  namespace: <namespace>
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-inference
  minReplicas: 2
  maxReplicas: 8
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      selectPolicy: Max
      policies:
      - {type: Pods, value: 1, periodSeconds: 60}
      - {type: Percent, value: 50, periodSeconds: 60}
    scaleDown:
      stabilizationWindowSeconds: 600
      selectPolicy: Min
      policies:
      - {type: Pods, value: 1, periodSeconds: 300}
  metrics:
  - type: Pods
    pods:
      metric:
        name: llm_queue_depth
      target:
        type: AverageValue
        averageValue: "4"

The HPA computes the desired replica count roughly as (currentReplicas × currentAverage) / targetAverage, then applies tolerance, missing‑metric handling, limits, policies, and stabilization windows. Because model cold‑start can be long, down‑scaling should be conservative.

Compute maxReplicas Capacity Boundary

The script below sums the allocatable GPUs across all nodes matching a label, subtracts a reserved amount, and prints the theoretical maximum number of replicas. It does not account for other workloads that may already have GPU requests.

#!/usr/bin/env bash
set -euo pipefail

LABEL='accelerator.example.com/gpu-family=a100'
RESERVED_GPU="<reserved‑gpu‑count>"

total="$(kubectl get nodes -n <namespace> -l "${LABEL}" -o json \
  | jq '[.items[].status.allocatable["nvidia.com/gpu"] | tonumber] | add // 0')"
if ! [[ "${RESERVED_GPU}" =~ ^[0-9]+$ ]]; then
  echo "RESERVED_GPU must be a non‑negative integer" >&2
  exit 2
fi
available=$(( total - RESERVED_GPU ))
(( available < 0 )) && available=0
printf 'allocatable_total=%d theoretical_max=%d
' "${total}" "${available}"

Final maxReplicas must also consider existing GPU requests, node affinity, taints, topology, model cache on disk, and memory. If node auto‑scaling is slower than the queue tolerance, keep a warm capacity buffer.

Canary Validation

Do not use production traffic to prove the HPA. The following script generates a controlled load against a canary service.

#!/usr/bin/env bash
set -euo pipefail

ENDPOINT="<canary‑service‑url>/v1/generate"
REQUESTS=40
CONCURRENCY=4
PAYLOAD='{"prompt":"health validation","max_tokens":8}'
seq "${REQUESTS}" | xargs -P "${CONCURRENCY}" -I{} \
  curl -fsS --max-time 30 \
    -H 'Content-Type: application/json' \
    -d "${PAYLOAD}" "${ENDPOINT}" >/dev/null

While the load runs, watch the HPA, deployment, pods, and events:

kubectl get hpa llm-inference -n <namespace> -w
kubectl get deployment llm-inference -n <namespace> -w
kubectl get pods -n <namespace> -l app=llm-inference -o wide
kubectl get events -n <namespace> \
  --sort-by='.metadata.creationTimestamp' | tail -n 50

If pods become Pending after the HPA adds replicas, the issue is capacity or scheduling; if pods are Running but not Ready, check model loading, probes, and storage; if the queue metric rises but the HPA does not react, verify the custom‑metrics API and HPA conditions.

Collect Troubleshooting Evidence

The read‑only script gathers the HPA definition, deployment status, pod list, recent events, custom‑metric values, and recent adapter logs.

#!/usr/bin/env bash
set -euo pipefail

NAMESPACE="<namespace>"
ADAPTER_NS="<adapter‑namespace>"
NAME="llm-inference"

kubectl get hpa "${NAME}" -n "${NAMESPACE}" -o yaml
kubectl get deployment "${NAME}" -n "${NAMESPACE}" -o wide
kubectl get pods -n "${NAMESPACE}" -l app="${NAME}" -o wide
kubectl get events -n "${NAMESPACE}" --sort-by='.metadata.creationTimestamp' | tail -n 80
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/${NAMESPACE}/pods/*/llm_queue_depth" -n "${NAMESPACE}" | jq .
kubectl logs -n "${ADAPTER_NS}" deployment/prometheus-adapter --since=30m | tail -n 200

Root‑cause analysis should follow a clear evidence chain: Prometheus has data → custom‑metrics API returns values → Adapter logs show successful mapping → HPA scales → pods acquire GPU and become ready. Any break in the chain points to the responsible component.

Metric Anomalies and Alerts

PromQL queries to monitor queue health and to fire alerts when the queue stays non‑zero at the HPA maximum:

sum by (namespace, pod) (llm_queue_depth{namespace="<namespace>"})
sum by (namespace, pod) (llm_inflight_requests{namespace="<namespace>"})
changes(llm_queue_depth{namespace="<namespace>"}[10m])

(kube_horizontalpodautoscaler_status_current_replicas{namespace="<namespace>",horizontalpodautoscaler="llm-inference"} ==
 kube_horizontalpodautoscaler_spec_max_replicas{namespace="<namespace>",horizontalpodautoscaler="llm-inference"})
and sum(llm_queue_depth{namespace="<namespace>"}) > 0

Do not mask a persistently growing queue by raising the target value; instead, fix the counting logic or increase capacity.

Emergency Freeze and Rollback

When a faulty metric causes runaway scaling, snapshot the HPA, patch both minReplicas and maxReplicas to a safe number, and later restore the original manifest.

# Snapshot
kubectl get hpa llm-inference -n <namespace> -o yaml > hpa.before.yaml
# Freeze
SAFE_REPLICAS="<safe‑replica‑count>"
kubectl patch hpa llm-inference -n <namespace> --type=merge \
  -p "{\"spec\":{\"minReplicas\":${SAFE_REPLICAS},\"maxReplicas\":${SAFE_REPLICAS}}}"
# Rollback HPA
kubectl apply -n <namespace> -f hpa.before.yaml --dry-run=server -o yaml > /tmp/hpa.rollback.rendered.yaml
kubectl diff -n <namespace> -f hpa.before.yaml || true
kubectl apply -n <namespace> -f hpa.before.yaml

Adapter upgrades should also be rolled back with helm rollback after a dry‑run and diff check, and the APIService must be verified.

# Adapter rollback
helm history prometheus-adapter -n <adapter‑namespace>
helm rollback prometheus-adapter <revision> -n <adapter‑namespace> --wait --timeout 10m
kubectl get apiservice v1beta1.custom.metrics.k8s.io -n <adapter‑namespace> -o wide

All permanent changes should be stored in version‑controlled manifests; use kubectl delete --dry-run=server -o yaml to preview deletions before removing resources.

Final checklist :

Queue counters decrement on success, timeout, and error paths.

Each pod’s time series is uniquely identifiable.

Prometheus, Adapter API, and HPA values are consistent.

Scaled‑out pods acquire GPU and become Ready within the allowed window.

maxReplicas respects node‑level GPU capacity.

Down‑scale window covers model cold‑start and traffic spikes.

Alerts distinguish metric faults, scheduling shortages, and model‑load failures.

Both Adapter and HPA have documented rollback procedures.