What Real‑World Kubernetes Lessons Reveal About Cloud‑Native Ops

1. Kubernetes Is Not Just Hype

After two years of moving a team from Ansible + Terraform to a fully Kubernetes‑based workflow, deployment frequency increased more than threefold while deployment errors became almost nonexistent. Visibility into operations improved, routine automation grew, and mean time to recovery after infrastructure incidents dropped dramatically.

2. Traefik + Cert‑Manager + External‑DNS Make HTTPS Easy

Using Traefik as the Ingress controller, Cert‑Manager for Let’s Encrypt certificates, and External‑DNS for automatic DNS record management creates a buttery‑smooth HTTP routing experience. Traefik’s rich metrics, minimal control‑plane footprint, and responsive development team make it a reliable edge proxy, while Cert‑Manager automates TLS lifecycle. External‑DNS, though less popular, is equally essential for keeping DNS records in sync.

3. Prometheus Is Powerful, Thanos Is Not Overkill

The team adopted Prometheus‑Operator as the primary metrics system, finding Prometheus to be the de‑facto standard. Adding Thanos early simplified cross‑region queries and reduced Prometheus resource consumption, even without a full master‑master high‑availability setup.

4. GitOps Is the Right Path

Implementing GitOps with tools like ArgoCD (preferred) and Flux turned infrastructure changes into version‑controlled, repeatable actions. A recent disaster‑recovery drill—accidental deletion of many namespaces—was resolved by running make apply from a bootstrap repo, with Velero handling stateful data such as cert‑manager certificates.

5. Build More Operators

Custom Operators, started with a single resource for the main network service, expanded to automate many platform components. While plain Kustomize + ArgoCD works for simple services, complex needs—such as creating AWS IAM roles via kiam or managing stateful migrations for Django apps—require operators. The team uses a thorough test suite to ensure reliability.

6. Secret Management Remains Tricky

Kubernetes Secrets work well at runtime, but storing raw secrets in Git is unsafe. The author created a custom EncryptedSecret type that encrypts values with AWS KMS; a controller decrypts them back to regular Secrets, and a CLI tool handles the encrypt‑edit‑decrypt cycle. Community operators based on Mozilla Sops offer similar workflows, but a fully auditable, version‑controlled secret lifecycle is still an open problem.

7. Native CI and Log Analysis Are Still Open Issues

While many CI systems (Jenkins, Concourse, Buildkite) can run on Kubernetes, truly native solutions are scarce. Jenkins X and Prow are close but bring complexity; Tekton Pipelines and Argo Workflows provide low‑level pipelines but lack easy developer exposure. For logging, Fluent Bit + Fluent d is the de‑facto stack, with Elasticsearch and Loki as storage back‑ends. Kibana’s advanced features require a commercial license, and Loki’s UI lacks robust permission controls, posing challenges for compliance audits.

Conclusion

Kubernetes is not a plug‑and‑play solution, but with careful engineering and a vibrant ecosystem it becomes an unparalleled platform. Investing time to understand each component—networking, ingress, observability, GitOps, operators, secrets, CI, and logging—helps avoid common pitfalls and paves the way to container‑centric success.