Scaling Non‑CPU‑Bound Apps with HPA Using cAdvisor Network Metrics

Background

Some business services are not CPU or memory sensitive and would benefit from Horizontal Pod Autoscaling (HPA) based on traffic metrics, but they lack Prometheus SDK instrumentation. cAdvisor can provide container network traffic metrics to achieve elastic scaling during peak and trough periods.

Solution Overview

cAdvisor, built into kubelet, exposes node and container resource statistics via the /metrics/cadvisor endpoint. It collects the following network counters: container_network_receive_bytes_total – total bytes received by the container.

container_network_transmit_bytes_total

– total bytes transmitted by the container.

Both metrics are counters that only increase, so they must be converted to a per‑second rate (e.g., using rate()) and optionally scaled to kilobytes.

Practical Steps

3.1 Install Prometheus‑related Plugins

Use Huawei Cloud CCE’s plugin marketplace to install kube‑prometheus‑stack, which already integrates cAdvisor metrics. After installation, you can view metrics in the Prometheus UI.

3.2 Configure Prometheus‑Adapter Metric Conversion Rules

Edit the adapter configmap:

kubectl -n monitoring edit configmap user-adapter-config

Add the following series queries and conversion rules:

- seriesQuery: 'container_network_receive_bytes_total{namespace!="",pod!=""}'
  seriesFilters: []
  resources:
    overrides:
      namespace:
        resource: namespace
      pod:
        resource: pod
  name:
    matches: container_(.*)_total
    as: "pod_${1}_per_second"
  metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[3m])) by (<<.GroupBy>>)/1000

- seriesQuery: 'container_network_transmit_bytes_total{namespace!="",pod!=""}'
  seriesFilters: []
  resources:
    overrides:
      namespace:
        resource: namespace
      pod:
        resource: pod
  name:
    matches: container_(.*)_total
    as: "pod_${1}_per_second"
  metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[3m])) by (<<.GroupBy>>)/1000

After editing, restart the custom-metrics-apiserver in the monitoring namespace.

The metricsQuery converts the cumulative counter to a per‑second rate and divides by 1000 to present kilobytes. The resources section maps Prometheus metrics to Kubernetes objects, while the name section renames metrics for readability.

3.3 Verify Custom Metric Availability

Query the metric via the Kubernetes API:

kubectl get --raw="/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/pod_network_receive_bytes_per_second" | jq

The response shows the metric values. The same metric can also be viewed in the CCE console.

3.4 Test HPA Elasticity

Create an HPA object that scales based on the custom network‑receive metric:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-app07
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app07
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: pod_network_receive_bytes_per_second
      target:
        type: AverageValue
        averageValue: 10

Generate traffic with a simple loop: while true; do curl clusterIP:port; done Watch the HPA status: kubectl get hpa hpa-app07 -w As the network‑receive metric rises, pods are added until the maximum replica count is reached; when traffic stops, the HPA scales back down to a single pod.

Supplementary Information

How to view container network traffic metrics in the CCE console.

Load‑level network traffic metric display.

Pod‑level network traffic metric display.

Alternative view via Cloud‑Native Observability → Monitoring → Dashboard → Pod view.

Link: https://bbs.huaweicloud.com/blogs/423255

© Huawei Cloud Community. All rights reserved.