How to Tackle Performance Optimization in Large‑Scale Kubernetes PaaS Platforms

Performance Optimization Challenges

The PaaS platform consists of four major subsystems: micro‑service governance, application scheduling & resource management, CI/CD pipeline, and cloud middleware services. Optimizing such a system is difficult because of many pain points:

100+ Git repositories; full build takes more than a day.

Complex deployment requiring 30+ VMs and 200+ processes.

Deep software stack and intricate network topology.

Cluster size of 5k‑10k nodes makes environment setup extremely hard.

Distributed operations prevent a single component from diagnosing bottlenecks.

Unable to trace latency and throughput of thousands of APIs across layers.

Developers often focus on features and overlook performance impact.

Optimization Analysis

The methodology is to decompose the large system into three loosely coupled dimensions and address each independently:

Control subsystem – command issuance and execution (Kubernetes), e.g., pod creation.

Data/traffic subsystem – container networking (flannel) and load balancing (ELB/kube‑proxy).

Monitoring subsystem – metric collection and alerting (Kafka, Hadoop).

Typical large‑scale deployment scenario:

Application package size: 400 MB

Application template size: 10 MB

1 000 nodes, each running a pod

10 package types, dependency depth 3, total network traffic 10 GB

Scheduling & resource management hosted on 3 VMs

Derived performance indicators:

Control: Kubernetes scheduling > 50 pods/s; repository download > 40 MB/s with 300 concurrent connections.

Data: Overlay network TCP overhead < 5 %.

Monitoring: Approx. 100 alerts/s (not involved in this scenario).

Testing & Tools

Prometheus is recommended for metric definition and collection. Backend programs embed the Prometheus SDK, expose an HTTP endpoint, and Prometheus scrapes the data into a time‑series database. This pull‑based model reduces measurement overhead.

Counter – monotonically increasing values (e.g., request count).

Gauge – values that can go up or down (e.g., CPU, memory).

Histogram – bucketed observations for latency distribution.

Summary – quantiles for request latency.

In the Kubernetes project, metrics are broken into five dimensions: verb, resource, client, content‑type, and HTTP code, enabling dashboards to pinpoint the most stressful request types.

Additional analysis methods:

Inspect pod phase counts during Kubernetes scheduling to locate the slowest step.

Use go pprof to identify CPU‑intensive functions.

Optimization Development

After bottlenecks are identified, developers improve code without changing functionality, typically by adding concurrency, introducing caching, or removing unnecessary steps.

Optimization Results

Control‑plane improvements achieved the following metrics (illustrated below):

Other subsystems, especially networking, have not yet been validated at the same scale; results are therefore indicative only.

Iterative Optimization Loop

Test to locate the bottleneck.

Modify code to eliminate the bottleneck.

Retest to verify the change and decide whether further refinement is needed.

Automation to Reduce Overhead

To cut down the time spent on building, environment setup, and reporting, the following automation techniques are employed:

Kubemark simulator – container‑based VM emulation achieving a 1:20 pressure ratio; 500 simulated VMs represent 10 000 nodes.

CI integration – automatically creates a performance‑optimization branch after a PR and triggers a fast build.

CD integration – snapshot‑based cluster provisioning for rapid test execution and report generation.

These practices enable early detection of performance regressions in CI, allowing developers to receive self‑service performance reports and address issues without heavy involvement from performance engineers.