How to Scale Kubernetes Clusters: Quotas, Kernel Tweaks, and Best Practices

1. Node Quotas and Kernel Parameter Adjustments

When scaling Kubernetes clusters on public clouds, you may encounter quota limits and need to increase them on the cloud platform. Quotas to increase include:

Number of virtual machines

Number of vCPUs

Number of internal IP addresses

Number of external IP addresses

Number of security groups

Number of route tables

Persistent storage size

Reference GCE master node configurations based on node count:

1-5 nodes: n1-standard-1

6-10 nodes: n1-standard-2

11-100 nodes: n1-standard-4

101-250 nodes: n1-standard-8

251-500 nodes: n1-standard-16

more than 500 nodes: n1-standard-32

Reference Alibaba Cloud configuration (kernel parameters):

# max-file sets the maximum number of file handles the system can open
fs.file-max=1000000

# ARP cache size
net.ipv4.neigh.default.gc_thresh1=1024
net.ipv4.neigh.default.gc_thresh2=4096
net.ipv4.neigh.default.gc_thresh3=8192

# Netfilter connection tracking limits
net.netfilter.nf_conntrack_max=10485760
net.core.netdev_max_backlog=10000
net.netfilter.nf_conntrack_tcp_timeout_established=300
net.netfilter.nf_conntrack_buckets=655360

# Inotify limits
fs.inotify.max_user_instances=524288
fs.inotify.max_user_watches=524288

2. Etcd Database

Deploy a highly available etcd cluster that can automatically scale; the common solution is to use the etcd operator, which simplifies management of stateful applications.

Key features of the etcd operator:

create/destroy: automatically provision and delete etcd clusters

resize: dynamically scale the cluster

backup: support backup and restore

upgrade: upgrade without service interruption

Additional recommendations:

Store etcd data on SSDs

Increase --quota-backend-bytes (default 2 GB)

Configure a dedicated etcd cluster for kube-apiserver events

3. Kube-APIServer Configuration

For node counts ≥ 3000, set:

--max-requests-inflight=3000
--max-mutating-requests-inflight=1000

For node counts between 1000 and 3000, set:

--max-requests-inflight=1500
--max-mutating-requests-inflight=500

Memory target (in MB) scales with node count:

--target-ram-mb=node_nums * 60

4. Pod Configuration

Best practices for running pods include setting resource requests and limits, especially for core add-on services:

spec.containers[].resources.limits.cpu
spec.containers[].resources.limits.memory
spec.containers[].resources.requests.cpu
spec.containers[].resources.requests.memory
spec.containers[].resources.limits.ephemeral-storage
spec.containers[].resources.requests.ephemeral-storage

Kubernetes classifies pods into QoS classes based on these settings: Guaranteed, Burstable, and BestEffort. When resources are scarce, the kubelet evicts pods in the order BestEffort > Burstable > Guaranteed.

Use nodeAffinity, podAffinity, and podAntiAffinity to spread critical workloads, e.g., for kube-dns:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      labelSelector:
        matchExpressions:
        - key: k8s-app
          operator: In
          values:
          - kube-dns
      topologyKey: kubernetes.io/hostname

Prefer managing containers with higher-level controllers such as Deployment, StatefulSet, DaemonSet, or Job.

Set kube-scheduler and controller-manager API QPS to 100 (default 50/20) and burst to 100 (default 30).