How I Cut Our K8s Cloud Bill by 60%: A Cost Optimization Case Study

Analysis: Where the Money Went

The author inherited a Kubernetes cluster with an annual cloud bill of ¥2.03 million. After performing an eBPF performance audit (revealing 5 issues invisible to traditional monitoring) and a security audit (finding 23 high-severity vulnerabilities caused by misconfigured Pod RBAC and image security), the author turned to resource utilization. A cost breakdown by environment showed:

Production: 32 nodes, ¥980K (48%)

Staging: 12 nodes, ¥420K (21%)

Test: 15 nodes, ¥380K (19%)

Dev/Other: 8 nodes, ¥250K (12%)

Staging and test environments together consumed ¥800K (40% of total) despite near-zero usage. Test environment QPS peaked at 127 only during a stress test; daily QPS was essentially zero. Staging had not received a deployment in two months.

Step 2: Node-Level Utilization Reveals Zombie Nodes

Per-node CPU analysis showed 18 of 50+ nodes averaged below 15% utilization. The most extreme case: an 8-core 32GB node running a single test-service Pod with 0 QPS, costing ¥23K/year. The service was created by an intern who had left six months earlier. Five such "zombie nodes" wasted ¥110K annually.

Step 3: Pod Resource Inflation

Kubernetes charges based on requested resources (requests), not actual usage. The author examined the top 20 Pods with the largest request-vs-actual gaps:

order-service: requested 8Gi memory, actual 380Mi (95% waste)

user-service: requested 4Gi, actual 210Mi (95% waste)

payment-service: requested 4Gi, actual 450Mi (89% waste)

report-service: requested 8Gi, actual 1.2Gi (85% waste)

The order-service developer admitted copying an 8Gi request from another service that actually needed it, while order-service traffic was only one-tenth. Cluster-wide similar over-provisioning wasted ~¥300K/year.

Resolution: Four Battles to Implement Savings

Battle 1: Scaling Down Test Environments

Proposal: auto-scale test environment to 2 nodes nights (8 PM–9 AM) and weekends. Test lead objected: "What if I need regression testing on weekends? Who's responsible if Monday morning scale-up delays us?" Author offered a manual webhook trigger. Deadlock lasted two weeks until leadership mandated a one-month trial. Result: only 2 manual scale-ups in a month; 13 nodes were indeed idle. Saved ¥200K, but took two weeks of persuasion.

Battle 2: Lowering Resource Requests with VPA

Plan: use Vertical Pod Autoscaler (VPA) to automatically adjust requests. Risk: VPA restarts Pods to apply new requests, potentially disrupting stateful services. Author tested in staging for a week, then proposed production canary. Architect blocked: "Have you assessed VPA restart risk? What if a stateful service restarts? Will reduced requests survive traffic spikes?" Compromise: run VPA in recommendation-only mode for a month. Data showed VPA recommended >50% memory request reduction for 60% of Pods. Architect still hesitated: "Recommendations are fine, but can you guarantee no issues?" Final compromise: manually lower requests for 10 stateless services, observe two weeks. Two services suffered OOMKill (auto-restarted within 10 seconds, still had headroom). Leadership warned: "Next incident stops the project." Saved ¥250K, but felt like walking a tightrope.

Battle 3: Cleaning Up Zombie Services

Script identified 32 services with zero QPS for 30+ days. Notified owners; responses varied: "Next quarter refactor, keep it," "Not my service," "Owner left," "Database has data, can't delete," "Forgot it existed, delete it." Only 15 confirmed deletable. Author adopted a gentle approach: scale all 32 to 0 replicas (retain config, release compute), delete after 3 months if untouched. Saved ¥100K; communication cost tripled technical effort.

Battle 4: Introducing Spot Instances

All workloads ran on on-demand instances. Spot instances cost 20–30% of on-demand; moving 30% of stateless workloads could save ¥150K/year. Risk: cloud provider can reclaim Spot anytime. Author prepared: only stateless services, PodDisruptionBudgets (PDB) to ensure minimum availability, multi-AZ distribution, Cluster Autoscaler to replace reclaimed nodes. SRE asked: "Pod rebuild time? If >30s, user impact?" Author tested: mostly 10–15s, worst-case 30–60s. SLA allows 43 min/month downtime (99.9%); Spot reclaims a few times a month, each <1 min, within SLA. Leadership approved for non-core services only. One month later: 3 reclaims, longest rebuild 45s, no user complaints. Saved ¥150K; author lost two nights' sleep before launch.

Results Summary

Zombie node/service cleanup: ¥210K saved; high communication overhead, some owners untraceable

Test environment off-hours scaling: ¥200K saved; initial test-team resistance

Resource request optimization: ¥250K saved; 2 OOMKills, leadership intervention

Spot instance adoption: ¥150K saved; pre-launch anxiety, 3 reclaim events

Dev environment downsizing: ¥120K saved; minimal friction

Other (log storage, image registry): ¥300K saved; low risk, smooth

Total: ¥1.23M saved (60% reduction), new annual bill ~¥800K

Lessons Learned

1. Cost Optimization Is an Organizational Problem, Not Just Technical

Technical savings list took one day; implementation took two months. Every optimization faced guardians of the status quo: developers fear OOM, testers fear delays, product managers fear future needs, SREs fear instability. The job is not to prove "this saves money" but "this saves money safely."

2. Start with Low-Risk Wins to Build Trust

Beginning with uncontroversial items (zombie nodes, dev environment, log storage) built credibility for harder battles (Spot, request reduction). Win small battles first to earn the right to fight big ones.

3. Data Beats Opinions

Claiming "test environment utilization is low" got pushback. Showing a 30-day CPU graph with nightly utilization below 5% ended the argument. Don't say "I think it's waste"; say "Here's the data, you decide."

4. Never Go All-In at Once

VPA: recommendation mode → 10-service manual trial → gradual rollout. Spot: 5% non-core → expand after validation. Every step has a rollback plan.

5. Cost Optimization Is Continuous, Not a Project

Today's optimal config becomes waste tomorrow due to new services, traffic changes, decommissions. Author now runs monthly cost analysis to catch new waste — more valuable than a one-time ¥1M cut.

Conclusion

Reducing the bill from ¥2.03M to ¥800K saved ¥1.23M — enough to hire two developers, buy more servers, or boost profit. But cost-cutting has boundaries: you cannot risk ¥2M to save ¥1M. Every cut requires judging: how much saved, what risk taken, can the risk be contained. The hardest part isn't technology; it's judgment and balance.