10 Hard‑Earned Lessons from 3 Years Managing Kubernetes Clusters

Background

Over the past three years the author has navigated the ups and downs of managing Kubernetes clusters, gaining deep insight into the technology and its surrounding ecosystem. This article distills the ten most valuable lessons learned, aimed at anyone from beginners to experienced operators.

Lesson 1: Use Managed Kubernetes in the Cloud

Unless you have extreme constraints, avoid managing the underlying Kubernetes infrastructure yourself. Debugging low‑level issues rarely adds business value. While understanding components like kube‑api, etcd, and kube‑proxy is useful, daily maintenance is better delegated to cloud providers (AWS, Azure, GCP, OVH, etc.). The author’s team uses AWS EKS.

Lesson 2: Deploy All Cluster‑Related Resources as Code

Never perform manual changes in the console, not even adding a simple label. Avoid the mindset of “quick fix in the UI, later update the code.” All cluster objects should be version‑controlled and applied automatically.

Lesson 3: Avoid Over‑reliance on Helm Charts You Can’t Fully Control

Helm charts are convenient, but you should understand every variable in values.yaml and avoid default values when possible. The author’s team prefers not to use Helm charts at all, falling back to raw templates if needed.

Lesson 4: Kubernetes Doesn’t Like “Lift‑and‑Shift”

Applications should be adapted to run on Kubernetes rather than forcing Kubernetes to fit legacy workloads. If you cannot refactor the application, consider keeping it on traditional VMs.

Lesson 5: Mesh or No Mesh?

Only install a service mesh if your workloads communicate with each other and you need mesh‑level security policies. Otherwise, skip it. The author notes that most mesh technologies are similar in functionality.

Lesson 6: Resist the Temptation to Use Too Many Tools

Kubernetes offers many auxiliary tools (Argo CD, Lens, k9s, KEDA, krew, kubectx, kubens, kail, etc.). Stick to kubectl for about 90 % of tasks; the author personally uses only kubectx, kubens, and k9s.

Lesson 7: Define Resource Limits for Pods

Set memory and CPU limits on every pod to prevent a single misbehaving workload from exhausting cluster resources. This also encourages careful review of Helm chart manifests.

Lesson 8: Embrace Stateless Design

Avoid storing data inside pods. If persistence is required, use network‑attached storage (e.g., EFS) rather than direct disk mounts, which are node‑specific and can cause data visibility issues across nodes.

Lesson 9: Configure Horizontal Pod Autoscaling (HPA)

To benefit from Kubernetes’ scaling capabilities, enable HPA on all applicable workloads, allowing the cluster to automatically adjust resources based on demand.

Lesson 10: Don’t Fear Change – Plan Regular Upgrades

Aim for three cluster upgrades per year, roughly every four months. Read release notes thoroughly and learn from others’ upgrade experiences. The author recommends staying on the version just before the latest, unless a security patch forces a newer release.

Wishing you a smooth and enjoyable Kubernetes journey!