Five Overlooked Runtime Risks When Deploying Large Language Models on Kubernetes

Problem Background

Running large‑model inference services on Kubernetes is now common because Kubernetes offers declarative configuration, auto‑scaling, rolling updates, and service discovery. However, many teams discover hidden runtime risks only after deployment in production, where high concurrency, long‑running requests, and resource contention expose stability issues.

Applicable Scenarios

Model inference services already deployed on Kubernetes.

Services experiencing stability problems in production (timeouts, pod restarts, resource spikes).

Teams planning to migrate model services to Kubernetes and want to avoid common pitfalls.

Core Knowledge

Kubernetes Pod Lifecycle

Pending → ContainerCreating → Running → Ready

Note that Pod ready only means the readiness probe succeeded; the model may still be loading.

Resource Limits

requests

and limits for CPU and memory are enforced by cgroups.

GPU memory is NOT limited by memory limits; it is managed by the device plugin and can cause OOM independently.

Health Checks

LivenessProbe determines if a container should be restarted.

ReadinessProbe determines if a pod can receive traffic.

Both can be httpGet, tcpSocket, or exec.

Network Model

Kubernetes uses CNI plugins (Calico, Flannel, Cilium). Service types (ClusterIP, NodePort, LoadBalancer) forward traffic without awareness of pod‑level load or queue length.

Logging & Monitoring

Only stdout / stderr are collected by default.

Framework logs may be written to files inside the container.

GPU metrics require DCGM Exporter or similar.

Runtime Risk 1 – Traffic Before Model Load

Symptom : Pod becomes Ready and receives requests while the model weight is still loading, causing “model not loaded” errors.

Root Cause : ReadinessProbe only checks the HTTP port, which is open before the model finishes loading.

Detection

Compare pod.status.conditions[?(@.type=="Ready")].lastTransitionTime with the timestamp of the model loaded log entry.

Run a quick request after the pod is Ready; a failure indicates the model is not ready.

Solution

Increase readinessProbe.initialDelaySeconds to cover model load time.

Use the framework’s health endpoint (e.g., /v1/models in vLLM) that reports model readiness.

Implement an exec probe that performs a real inference request.

Combine startupProbe (to wait for model load) with readinessProbe.

Production Recommendations

Prefer startupProbe + readinessProbe for slow‑loading models.

Set readinessProbe to the framework’s model‑list API.

Configure reasonable failureThreshold to avoid premature restarts.

Runtime Risk 2 – GPU Memory OOM and Infinite Restarts

Symptom : Pods are killed repeatedly with OOMKilled despite generous memory limits.

Root Cause : GPU memory is not limited by memory limits; large batch sizes or multi‑process inference can exceed the physical GPU memory.

Detection

Check pod restart count and lastState.terminated.reason for OOMKilled.

Inspect nvidia-smi or DCGM metrics for high memory usage.

Search container logs for “out of memory” messages.

Solution

Reduce batch size or concurrency ( --max-num-seqs, --max-model-len).

Enable GPU memory paging if supported (e.g., vLLM --swap-space).

Use multi‑GPU tensor parallelism.

Set GPU memory reservation flags (e.g., PYTORCH_CUDA_ALLOC_CONF).

Deploy DCGM Exporter and set alerts for memory usage >90%.

Production Recommendations

Monitor GPU memory with DCGM Exporter.

Pre‑estimate GPU memory needs per model and batch size.

Test high‑concurrency scenarios before release.

Runtime Risk 3 – LivenessProbe Mis‑Kills Long Inference

Symptom : Long requests are aborted because the LivenessProbe times out and restarts the pod.

Root Cause : LivenessProbe timeout is shorter than the maximum inference latency.

Detection

Inspect kubectl describe pod for events containing “Liveness probe failed” and “Killing”.

Compare P99 request latency with total LivenessProbe timeout ( periodSeconds × failureThreshold).

Solution

Increase LivenessProbe timeoutSeconds, periodSeconds, and failureThreshold so total timeout exceeds P99 latency.

Use a dedicated health endpoint that does not invoke inference (e.g., /health).

Switch to a TCP probe.

Implement an exec probe that checks inference queue length.

Disable LivenessProbe if the service rarely hangs, relying on external monitoring.

Production Recommendations

Ensure LivenessProbe total timeout > P99 inference latency.

Prefer independent health checks or TCP probes.

Monitor LivenessProbe failures and set alerts.

Runtime Risk 4 – Slow HPA Scaling Causes Traffic Spikes

Symptom : Sudden traffic bursts lead to many request timeouts; HPA only scales after several minutes.

Root Cause : HPA relies on CPU/memory metrics, which do not reflect GPU‑bound inference load; metric collection, scheduling, image pull, and model load add latency.

Detection

Check HPA events and timestamps versus traffic peaks.

Compare request latency (e.g., P99) with HPA scaling timestamps.

Monitor vllm_num_requests_waiting queue length.

Solution

Use custom metrics (e.g., inference queue length or request latency) for HPA.

Set higher minReplicas or schedule pre‑scale‑up via CronJob.

Adopt KEDA or event‑driven scaling based on Prometheus metrics.

Pre‑load models into container images or use InitContainers to cache models.

Production Recommendations

Configure HPA to scale on vllm_num_requests_waiting or latency.

Set realistic minReplicas based on peak traffic.

Ensure Cluster Autoscaler can provision nodes quickly.

Runtime Risk 5 – Missing Framework Logs Hinder Debugging

Symptom : kubectl logs shows no useful output; errors are hidden in internal log files.

Root Cause : Inference frameworks write logs to files inside the container instead of stdout / stderr.

Detection

Exec into the pod and locate log files (e.g., /tmp/vllm.log).

Diff kubectl logs output with internal log file content.

Review logging collector configuration (Fluentd, Filebeat).

Solution

Configure the framework to output logs to stdout (e.g., VLLM_LOG_TO_STDOUT=true).

Deploy a sidecar container (Fluentd) that tails internal log files.

Mount a hostPath or emptyDir volume for persistent logs.

Use centralized logging (Loki, ELK) to collect both stdout and file logs.

Production Recommendations

Prefer stdout logging for all inference services.

Deploy a centralized log collector (Loki/Promtail or ELK).

Set log retention policies to avoid disk exhaustion.

Monitoring Checklist

Pod Status

Pod Ready time, restart count, last termination reason.

Startup/Readiness/Liveness probe failures.

GPU Metrics (DCGM Exporter)

Memory usage ratio, utilization, temperature, power.

Inference Performance

QPS, latency percentiles, queue length, success rate.

HPA Metrics

Current and desired replica counts, scaling events.

Fault Drills

Simulate GPU OOM by increasing batch size; verify pod restart and alert.

Trigger LivenessProbe timeout with a long request; verify mis‑kill and fix.

Generate traffic spike; observe HPA scaling latency and adjust custom metrics.

Best‑Practice Summary

Set appropriate CPU/GPU resources; remember GPU memory is unmanaged by cgroups.

Use startupProbe + readinessProbe that checks actual model load.

Configure LivenessProbe timeout to exceed P99 latency or use independent health checks.

Scale on inference‑specific metrics (queue length, latency) instead of CPU.

Monitor GPU memory with DCGM Exporter and set alerts at 90% usage.

Ensure all framework logs are sent to stdout or collected by a sidecar.