How to Build an Elastic Auto‑Scaling Cloud‑Native Application

Introduction

During a major sales event the team’s e‑commerce system saw traffic rise to 15 × the normal level, response time jumped from 200 ms to 8 s, and only manual scaling kept the service alive. The root cause was a lack of true elastic scaling, prompting a complete redesign.

What True Elastic Scaling Means

Elastic scaling is more than “adding machines automatically”. It comprises three layers:

Horizontal scaling – dynamically adjusting the number of service instances.

Vertical scaling – changing CPU and memory limits of a pod; Kubernetes 1.31+ supports In‑Place Pod Resize.

Architectural resilience – the system can degrade gracefully when components fail.

After the redesign the system runs 12 pods in normal operation and automatically expands to 80 pods during peak traffic, completing the scale‑up in under 90 seconds and scaling down after 20 minutes.

Overall Architecture

The design follows a “layered decoupling, independent scaling” principle. Each layer can scale independently, business services communicate via a message queue, and the traffic‑entry layer provides unified rate‑limiting and routing.

Core Component Breakdown

Traffic‑Entry Layer

The team replaced Nginx Ingress with Envoy Gateway because it natively supports request‑header based traffic splitting, canary releases, and deep xDS integration. Paired with the cloud provider’s ALB, the entry layer itself is elastic.

A key configuration is request queuing: when back‑ends are saturated the gateway queues requests instead of returning 503, only responding with 503 when the queue is full. This buys a valuable scaling window during the first 30 seconds of a traffic spike.

Business Service Layer

Each microservice runs in a Kubernetes pod packaged as a container image. Startup latency is the lower bound for scaling, so the team applied three optimisations:

Switched base images to Chainguard distroless, reducing size from 800 MB to 120 MB.

Compiled Java services to GraalVM native images, cutting startup from 12 s to 0.8 s.

Advanced the readiness probe for Go services so it passes in 3 seconds.

Data Layer

The database choice is TiDB for mature horizontal scaling, with Redis Cluster (six shards) for caching and a cloud‑provider S3‑compatible object store.

A painful lesson: the database connection pool must scale with the pod count. Initially each pod kept 50 connections; when pods grew from 12 to 80 the database hit 4 000 connections and saturated. Switching to PgBouncer with unlimited connections from pods to PgBouncer and a fixed 200 connections from PgBouncer to the database resolved the issue.

Elastic Scaling Strategy Selection and Implementation

The scaling strategy is the “brain” of the system; a wrong choice wastes resources.

4.1 Replacing Native HPA with KEDA

Kubernetes HPA only reacts to CPU and memory, which is insufficient. The team adopted KEDA (Kubernetes Event‑Driven Autoscaling), which can trigger scaling from dozens of event sources:

Kafka consumer lag > 10 000 messages.

HTTP P99 latency > 500 ms.

Scheduled jobs – pre‑scale at 9 am for weekday peaks.

Example KEDA configuration for the order service: three triggers – CPU > 60 %, QPS > 500 per pod per second, or P99 latency > 300 ms – any one triggers a scale‑up; scale‑down occurs only after all three metrics fall below thresholds for five minutes.

4.2 Predictive Scaling

Reactive scaling suffers from the delay between metric breach and new pods becoming ready. The team added a predictive scaling component that uses Prometheus recording rules to compute the average and standard deviation of traffic for the same time window over the past four weeks, then pre‑scales 15 minutes ahead. The simple statistical method achieves about 85 % accuracy without complex machine‑learning models.

4.3 The Art of Scale‑Down

Scale‑down is harder than scale‑up. The team settled on these parameters:

Cool‑down period of 5 minutes (KEDA default is 2 minutes, HPA default is 5 minutes).

Maximum reduction per step limited to 20 % of current instances.

Large‑scale reductions only during low‑traffic windows (e.g., 1 am–6 am).

Observability: The Eyes of Scaling

Without observability, scaling is blind. The stack consists of:

Metrics : Prometheus + Grafana Mimir for long‑term storage and global queries. Key metrics include per‑service QPS, latency percentiles (P50/P90/P99), pod count, and scaling event timelines.

Logs : OpenTelemetry Collector forwards logs to ClickHouse, which outperforms Elasticsearch by 5–10× in query speed and costs one‑third as much.

Traces : OpenTelemetry SDK embedded in services sends traces to Grafana Tempo. Traces helped identify a case where a new pod’s local cache was cold, causing latency spikes after a scale‑up.

A dedicated “elastic scaling dashboard” in Grafana overlays traffic curves, pod counts, scaling events, and resource utilization on a single timeline.

Common Pitfalls in Practice

Several issues the team encountered that are rarely documented:

Pitfall 1: JVM Warm‑up – Java services experience high latency for the first few hundred requests before JIT compilation finishes, potentially triggering unnecessary scale‑ups. Solutions: use GraalVM AOT compilation or add a warm‑up script to the readiness probe.

Pitfall 2: Stateful Service Scaling – WebSocket services cannot be simply reduced; they require graceful shutdown signals and client reconnection handling before pods are terminated.

Pitfall 3: Resource Fragmentation – Frequent scaling leaves nodes with small leftover resources (e.g., 1.5 CPU, 3 GB RAM) that cannot host a new pod needing 2 CPU. Using Karpenter instead of the default Cluster Autoscaler selects appropriate instance types and can reduce costs by 40‑60 %.

Pitfall 4: Database Bottleneck – Scaling the service layer to 100 pods while the database remains a single primary node creates a write bottleneck. Options include distributed databases (TiDB, CockroachDB) or read/write splitting with sharding; there is no one‑size‑fits‑all solution.

Conclusion

Effective elastic scaling hinges on three capabilities: rapid perception (observability), rapid decision (scaling policy), and rapid execution (containerization with fast start‑up). Weakness in any area limits overall elasticity.

Quantitative results: before the redesign, a major sales event required three days of manual scaling preparation and three‑fold resource over‑provision, costing roughly ¥250 000. After automation, daily resource utilization rose from 15 % to 55 %, peak resource cost during the event dropped by 60 %, and annual cloud spend saved nearly ¥1 000 000.

Elastic scaling is an ongoing effort; continuous adjustment of policies and parameters is required, but with a solid foundation the team no longer faces all‑night manual scaling before each promotion.