How I Cut $280K from a K8s Bill — And Why People Were the Real Challenge

Discovering the Waste

The author inherited a Kubernetes cluster with an annual cloud bill of ¥2.03 million (≈$280K). A cost breakdown showed:

Environment      Nodes  Annual Cost (¥10K)  Percentage
Production       32     98                  48%
Pre-production   12     42                  21%
Test             15     38                  19%
Dev/Other        8      25                  12%

Pre-production and test together consumed 40% of spend despite near-zero usage: test environment QPS peaked at 127 only during a load test; pre-production had not received a deployment in two months.

Node-Level Analysis

Examining 50+ nodes revealed 18 with average CPU utilization below 15%. The worst offender: an 8-core/32GB node running a single test-service Pod with 0 QPS, costing ¥23K/year. The service was created by an intern who had left six months earlier. Five such "zombie nodes" wasted ¥110K annually.

Pod Resource Inflation

Kubernetes charges for requested resources (requests), not actual usage. The author pulled the top 20 pods by request/actual gap:

Service          Requested Memory  Actual Usage  Waste
order-service    8Gi               380Mi         95%
user-service     4Gi               210Mi         95%
payment-service  4Gi               450Mi         89%
report-service   8Gi               1.2Gi         85%
...

The order-service developer admitted copying an 8Gi request from another service that actually needed it; order-service traffic was only one-tenth. Cluster-wide similar over-provisioning wasted ~¥300K/year.

Four Battles to Implement Savings

Battle 1: Test Environment Scale-Down

Proposal: auto-scale test environment to 2 nodes nights (8 PM–9 AM) and weekends. Test lead objected: "What if I need regression testing on weekends?" and "Monday morning scale-up delays our progress." A manual webhook trigger was offered but deemed "too much hassle." After a two-week standoff, leadership mandated a one-month trial. Result: only 2 manual scale-ups needed; 13 nodes were idle the rest of the time. Saved ¥200K, but took two weeks of persuasion.

Battle 2: Lowering Resource Requests with VPA

Vertical Pod Autoscaler (VPA) can automatically adjust requests, but it restarts pods. Architect raised risks: stateful service restarts, traffic spikes after reduction. Compromise: run VPA in recommendation mode for a month. Data showed 60% of pods could cut memory requests by >50%. Architect still hesitated: "Recommendations are not guarantees." Final agreement: manually lower requests for 10 stateless services, observe two weeks. Two services hit OOMKill (auto-restarted in <10 s, still had headroom). Leadership warned: another incident stops the project. Saved ¥250K, but felt like walking a tightrope.

Note: VPA works well for stateless services; stateful services should use HPA + manual request tuning to avoid restart-induced state loss.

Battle 3: Cleaning Zombie Services

Script identified 32 services with zero QPS for 30+ days. Notified owners; replies varied: "keeping for next quarter rewrite," "not my service anymore," "owner left," "database has data," "forgot it existed." Only 15 confirmed deletable. Compromise: scale all 32 to 0 replicas (retain config, release compute), delete after 3 months if untouched. Saved ¥100K; communication cost tripled technical effort.

Battle 4: Adopting Spot Instances

Spot instances cost 20–30% of on-demand. Moving 30% of stateless workloads could save ¥150K/year. Risks: cloud can reclaim anytime. Mitigations: only stateless services, PodDisruptionBudgets, multi-AZ distribution, Cluster Autoscaler for replacement. SRE asked: "Pod rebuild time? Impact if >30 s?" Author tested: mostly 10–15 s, worst case 30–60 s. SLA allows 43 min/month downtime (99.9%); expected Spot interruptions fit. Leadership approved for non-core services only. One month in: 3 reclaims, longest rebuild 45 s, no user complaints. Saved ¥150K; author lost two nights' sleep before launch.

Final Results

Optimization                     Annual Savings (¥10K)  Cost/Trade-off
Zombie node/service cleanup      21                     High communication overhead, some owners untraceable
Test env off-hours scale-down    20                     Test team initial resistance
Resource request tuning          25                     2 OOMKills, one leadership escalation
Spot instance adoption           15                     Pre-launch anxiety, 3 reclaim events
Dev environment downsizing       12                     Minimal friction
Other (logs, image registry)     30                     Low risk, smooth
Total                            123                    —

Bill dropped from ¥2.03M to ¥800K.

Key Takeaways

Cost optimization is an organizational problem, not a technical one. Every saving has a constituency protecting its buffer: devs fear OOM, test fears delays, product fears future needs, SRE fears instability. The job is proving "safe AND cheaper," not just "cheaper."

Start with low-risk wins to build trust. Clean zombie nodes, downsize dev, optimize storage first. Use those uncontroversial savings as credibility for harder changes.

Data beats opinions. A chart showing test env CPU <5% every night silenced the test lead instantly. Show the numbers; don't argue feelings.

Never go all-in at once. VPA ran in recommendation mode for a month, then 10-service pilot, then rollout. Spot started at 5% non-core, then expanded. Every step has a rollback plan.

Cost optimization is continuous, not a project. New services, traffic shifts, decommissions constantly change the landscape. The author now runs monthly cost analyses — ongoing vigilance beats one-off cuts.

Closing Thought

Operations value isn't just stability; under stability, saving money is equally valuable — perhaps more. ¥1.23M saved could hire two developers, buy more servers, or become profit. But every cut has a boundary: never risk ¥2M to save ¥1M. The hardest part isn't the technology; it's judgment and balance.

Source: https://zhuanlan.zhihu.com/p/2037196631498601353