10 Hard‑Earned Kubernetes Lessons Every Engineer Should Know

Background

The author, Herve Khg, a Multi‑Cloud (Azure/AWS) Systems and DevOps Engineer in France, reflects on three years of hands‑on Kubernetes administration. He distills the most valuable insights he gained into ten concrete lessons for anyone managing Kubernetes clusters.

Lesson 1: Use Managed Kubernetes in the Cloud

Unless you have extreme constraints, avoid managing the underlying Kubernetes infrastructure yourself. Maintaining components such as the kube‑api, kube‑apiserver, kubelet, etcd, and kube‑proxy consumes time without adding business value. Delegate these low‑level tasks to cloud providers (AWS, Azure, GCP, OVH, etc.). The author’s team uses AWS EKS.

Lesson 2: Deploy All Kubernetes‑Related Infrastructure as Code

Never make manual changes in the console, even simple label edits. Avoid the mindset of “quickly fix it in the console, then update the code later.” All cluster resources should be defined and version‑controlled in code.

Lesson 3: Don’t Over‑rely on Helm Charts You Can’t Fully Control

Helm charts are convenient, but you must understand every variable in values.yaml and avoid default values that could break your deployment. In the author’s organization, Helm charts are not used unless the template is inspected and customized.

Lesson 4: Kubernetes Doesn’t Like “Lift‑and‑Shift”

Instead of forcing Kubernetes to fit legacy applications, adapt the applications to Kubernetes. If you cannot refactor the apps, it may be better to keep them on traditional VMs.

Lesson 5: Mesh or No Mesh?

Only install a service mesh if it’s truly needed. Ask two questions: (1) Do the applications in the cluster communicate with each other? (2) Do they require inter‑service security policies? If both answers are yes, a mesh may be useful; otherwise, skip it.

Lesson 6: Avoid Using Too Many Tools

Kubernetes offers many auxiliary tools (ArgoCD, Lens, k9s, KEDA, krew, kubectx, kubens, kail, etc.). Collecting them like stamps adds complexity. In most cases, kubectl satisfies about 90% of needs. The author personally uses only kubectx, kubens, and k9s.

Lesson 7: Define Resource Limits for Pods

Set explicit CPU and memory limits for every pod to prevent a single misbehaving application from exhausting cluster resources and causing cascading failures. This also encourages careful review of Helm‑generated manifests.

Lesson 8: Embrace Stateless Design

Avoid storing data inside pods. If persistence is required, use network‑attached storage (e.g., EFS) rather than mounting local disks, which are node‑specific and can cause data visibility issues across nodes.

Lesson 9: Configure Horizontal Pod Autoscaling (HPA)

To reap Kubernetes’ scaling benefits, enable HPA on all application workloads. This automatically adjusts resource usage based on demand, addressing a common limitation of Helm‑based deployments.

Lesson 10: Don’t Fear Change – Plan Regular Upgrades

Aim for three cluster upgrades per year, roughly every four months. Read release notes thoroughly and learn from others’ upgrade experiences. The author’s practice is to stay on the version just before the latest, unless a security patch forces an immediate update.

Happy Kubernetes journey!