Optimizing Kubernetes Cluster Resource Utilization: Pod Compression, Node Overcommit, HPAPlus, and VPAPlus

The recent Cloud+ Community technical salon "Efficient Intelligent Operations" presented a deep dive into improving Kubernetes cluster resource utilization. The talk, delivered by senior Tencent Cloud engineer Zhuang Pengrui, covered the challenges of low utilization and introduced a suite of optimization techniques.

Why cluster resource utilization is often insufficient: (1) Node resource fragmentation – even when a node appears 90% full, leftover fragmented resources prevent new Pods from being scheduled. (2) Unreasonable Pod resource configuration – users often over‑estimate requests and set limits that do not match actual needs. (3) Improper Workload/HPA replica settings – replica counts are not aligned with real workload demand. (4) Business idle periods – certain workloads (e.g., games) have predictable peak and off‑peak times, leading to under‑utilization during low‑traffic windows.

Solution overview: The team introduced five key optimization points: Pod compression, Node over‑commit, HPAPlus, VPAPlus, and dynamic scheduling. These measures collectively compress resource requests, safely over‑sell node capacity, and enhance horizontal and vertical autoscaling.

1. Pod Compression – Requests are automatically reduced while keeping limits unchanged. This is achieved via a Mutating Admission Controller that intercepts Pod creation, applies a configurable ratio to the request fields, and stores the modified object back to etcd. The process expands the number of Pods that can be scheduled without exceeding node capacity.

2. Node Over‑Commit – By analyzing historical node usage via Prometheus, an over‑commit ratio is calculated for nodes with low actual utilization. A Mutating Admission Controller then adjusts the node’s allocatable resources during the kubelet’s periodic status patch, allowing more Pods to be scheduled on the same physical node.

3. HPAPlus – A standalone component that extracts the native HPA logic from the controller‑manager and runs it independently. It gathers metrics from three sources: the built‑in metrics‑server, Prometheus (for CPU, memory, and custom metrics), and a custom adapter for business‑specific metrics (e.g., connections, TPS). HPAPlus supports limits‑based scaling, per‑HPA goroutine execution, configurable sync intervals, suppression windows, Cron‑based scaling, and dynamic minimum replica calculation based on historical usage.

4. VPAPlus – Extends the vertical pod autoscaler by modifying core Kubernetes code. Changes include: (a) relaxing kube‑apiserver validation to allow pod resource updates, (b) adding resource‑update logic to kubelet’s sync loop, (c) handling cgroup memory and swap limits, and (d) implementing a recommender that builds a histogram‑based checkpoint from historical usage to produce 95 %, 90 %, and 50 % percentile recommendations. An update‑controller then applies these recommendations to Pods. VPAPlus also offers dynamic cgroup adjustments, request/limit ratio configuration, resource caps to avoid node overload, and controlled update frequencies.

Q&A – When core components are modified, the changes are packaged as plugins to avoid interfering with future Kubernetes upgrades. Internal version management ensures that updated kubelet or kube‑apiserver code is merged into a unified Tencent Cloud version.

Speaker Introduction – Zhuang Pengrui, senior engineer at Tencent Cloud, with experience building Kubernetes platforms at Vipshop, Vivo, and Tencent. He focuses on design and development of internal Kubernetes solutions.