Hidden Costs of Running Multiple Kubernetes Nodes: Resource Reservations and Cloud Provider Differences
This article explains how Kubernetes nodes reserve CPU and memory for the operating system, system daemons, and eviction thresholds, showing concrete examples across AWS EKS, Azure AKS, and Google GKE, and provides tools to inspect and calculate these hidden resource costs.
Running multiple Kubernetes cluster nodes incurs hidden costs because not all CPU and memory on a node are available for Pods.
Resources on a node are divided among the operating system, system daemons (kubelet, CNI, CRI, CSI), Pods, and eviction thresholds.
Example with a 1 GiB/1 vCPU node shows that kubelet and the OS reserve 255 MiB memory and 60 m CPU, plus a 100 MiB eviction threshold, leaving about 25 % of memory and 6 % of CPU unusable.
On AWS EKS using an m5.large (2 vCPU, 8 GiB) instance, the reserved resources are 574 MiB memory and 70 m CPU, allowing roughly 93 % of memory to be used for Pods.
CPU reservation formulas across cloud providers generally reserve 6 % of the first core, 1 % of the next core (up to 2 cores), 0.5 % of the next two cores (up to 4 cores), and 0.25 % for each core beyond four.
Memory reservation varies: Azure reserves 25 % of the first 4 GiB, 20 % of the next 4 GiB (max 8 GiB), 10 % of the next 8 GiB (max 16 GiB), 6 % of the next 112 GiB (max 128 GiB), and 2 % beyond 128 GiB; GKE follows the same rules except the eviction threshold is 100 MiB (AKS uses 750 MiB).
For EKS the memory reservation formula is 255MiB + (11MiB * MAX_NUMBER OF POD) , which can limit the number of Pods unless the VPC‑CNI prefix allocation mode is enabled; with prefix allocation, up to 110 Pods reserve 1.4 GiB memory and 70 m CPU.
On GKE with an n1‑standard‑2 (2 vCPU, 7.5 GiB) instance, kubelet reserves about 1.7 GiB memory and 70 m CPU, meaning roughly 23 % of the instance memory is unavailable to Pods.
If the instance costs $48.54 per month, you will spend $11.16 on kubelet overhead.
A simple tool ( kubernetes-resource-inspector ) can inspect kubelet configuration, and an instance calculator helps evaluate suitable node sizes and costs.
Additional resources include links to the Kubernetes instance calculator, documentation on allocatable resources for various cloud providers, and guides for enabling prefix assignment in EKS and VPC CNI.
System Architect Go
Programming, architecture, application development, message queues, middleware, databases, containerization, big data, image processing, machine learning, AI, personal growth.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.