Kubernetes Ops Cheat Sheet: From Daily Troubleshooting to Production‑Ready Architecture

Why memorizing commands isn’t enough – Effective Kubernetes operations require understanding four layers: the command layer (get, describe, logs, exec, etc.), the principle layer (how Pods, Deployments, Services work), the architecture layer (API Server, etcd, Scheduler, Controller Manager, kubelet, CNI, CSI, CoreDNS, Ingress), and the engineering layer (scalable, observable, rollback‑ready practices).

Core control‑plane components – API Server (central entry, audit, rate‑limit), etcd (metadata store, consistency), Scheduler (node selection), Controller Manager (state convergence), kubelet (node agent), kube‑proxy (Service routing), CNI (network), CSI (storage), CoreDNS (service discovery), Ingress Controller (L7 routing). Knowing each component’s operational concerns (permissions, latency, failures) guides troubleshooting.

Typical request flow –

user → LB/Ingress → Service → Pod → container process → storage / downstream service

. Follow this chain when diagnosing issues, avoiding premature exec into containers.

kubectl syntax – General form: kubectl [command] [TYPE] [NAME] [flags]. Common examples: kubectl get pods -n prod – list Pods in a namespace. kubectl get pods -A – list all Pods. kubectl get pods -o wide -n prod – show IP, node, image. kubectl get deploy,svc,ing -n prod – view multiple resources. kubectl get events -A --sort-by=.metadata.creationTimestamp – chronological events.

High‑frequency commands kubectl describe pod <pod> -n prod – detailed state, events, scheduling, probes. kubectl logs <pod> -n prod – container logs; add -f for streaming. kubectl logs -p <pod> -n prod – previous container logs (useful for CrashLoopBackOff). kubectl exec -it <pod> -n prod -- /bin/sh – interactive shell (avoid as first step). kubectl cp prod/api-xxx:/tmp/heap.bin ./heap.bin – copy files for analysis. kubectl port-forward svc/order-service -n prod 8080:80 – local access to a Service.

Production deployment workflow – Use declarative kubectl apply -f deployment.yaml, kubectl diff -f for pre‑flight diff, and kubectl rollout status to monitor rolling updates. A robust Deployment includes:

Replica count and revision history.

RollingUpdate strategy with maxUnavailable: 0 and maxSurge: 25% for critical APIs.

Readiness and liveness probes, startup probes, graceful termination, preStop hooks.

Pod anti‑affinity and topology spread constraints to avoid single‑node or single‑zone failures.

Pod troubleshooting – Follow the standard path:

Check Pod status ( kubectl get pod).

Inspect events via kubectl describe pod.

Read logs ( kubectl logs, kubectl logs -p).

If needed, exec into the container to verify env vars, DNS, or run health checks.

Network & Service diagnostics – Verify Service selectors, Endpoints, and Ingress rules. Use kubectl exec to run nslookup or curl from a Pod. For Service timeouts, ensure selector matches Pod labels, Pods are Ready, and targetPort aligns with container ports.

Storage management – PVC is a claim, PV is the provisioned volume, StorageClass defines the provisioner. Common commands: kubectl get pvc,pv -n prod, kubectl describe pvc,

kubectl patch pvc <name> -n prod -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'

for online expansion. Understand common failure modes such as Pending PVC, ContainerCreating, or data loss due to PV reclaim policies.

Node maintenance – Drain safely: kubectl cordon <node>,

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

, then kubectl uncordon <node>. Watch for PDB constraints, DaemonSets, local storage, or static Pods that can block draining.

RBAC best practices – Create dedicated ServiceAccounts per application, grant minimal Role permissions within the namespace, avoid using the default ServiceAccount, and restrict high‑privilege verbs (secrets, pods/exec, pods/attach). Example RoleBinding ties a ServiceAccount to a Role that only allows get on ConfigMaps.

High‑concurrency governance – Combine resource requests/limits, HPA (CPU & memory metrics), PDB, pod anti‑affinity, topology spread, readiness probes, graceful termination, and Cluster Autoscaler. A production HPA example uses aggressive scale‑up policies (100% per minute) and conservative scale‑down (20% per minute) with both CPU and memory utilization targets.

Observability stack – Metrics via Prometheus/Grafana, logs via Loki or ELK, tracing via Jaeger/Tempo, events via Kubernetes Events, alerts via Alertmanager. Relying solely on kubectl logs is insufficient; integrate these systems for proactive detection.

GitOps workflow – Commit code → CI builds image → security scan → Helm/Kustomize update → GitOps controller sync → monitor → rollback if needed. Commands like kubectl diff and kubectl apply are used for emergency fixes but must be reconciled back to Git.

Common anti‑patterns – Assuming Running means healthy, using only CPU‑based HPA, omitting PDB, not setting requests, over‑relying on kubectl exec, and performing manual “kubectl edit” changes without version control.

On‑call command checklist – Resource view ( kubectl get pods -A, kubectl top pod -A), rollout commands ( kubectl rollout status, kubectl rollout undo), pod debugging ( kubectl describe pod, kubectl logs, kubectl exec), network checks ( kubectl get svc,endpoints,ing, kubectl exec … nslookup), node maintenance ( kubectl cordon/drain/uncordon).

Key takeaways – Mastering kubectl is only the start; a mature Kubernetes operator links command output to control‑plane theory, service architecture, and production engineering practices such as scaling, observability, and automated GitOps pipelines.