Should You Migrate All Internal PaaS to Kubernetes? A Decision Framework with Coexistence Cost Analysis

The article opens with a common review-meeting blind spot: migration plans typically count only the "move" cost (app refactoring person-days, new cluster nodes) but ignore the coexistence period — the months or years when old and new platforms run side-by-side, requiring dual monitoring, alerting, on-call, and runbooks.

AWS Elastic Beanstalk as a Case Study

AWS added an EKS-backed Cluster Mode to Elastic Beanstalk while retaining the original EC2-based Standard Mode. The developer experience layer (app, version, environment, config concepts) stayed identical; only the orchestration base gained a new option. This illustrates the core architectural question: are the experience layer and the base one-to-one bound? If bound, swapping the base forces developers to relearn and pipelines to be rewritten; if decoupled, the base can be swapped per scenario with near-zero developer impact.

Four Prerequisites Before Deciding

The author recommends a go/no-go checklist before debating Kubernetes merits:

Who owns Kubernetes itself? Community releases ~3 minor versions/year with ~14-month patch support. Clusters falling outside the support window need control-plane upgrades, CNI/CSI troubleshooting, cert rotation — someone must be on-call. If the platform team lacks this skill, the only safe paths are managed services (accepting their constraints) or not migrating.

How many existing apps are truly stateless? Beanstalk's Cluster Mode mandates stateless, interchangeable replicas, ephemeral local storage only. The author suggests tagging every app into four buckets: Stateless (direct containerize), Refactorable (externalize sessions, move files to object storage, add distributed locks for cron), Strongly Stateful (databases, message queues, core ledger), Special Runtime (Windows, legacy middleware, hardware/license-bound). The first two buckets' share determines how far "unified base" can go.

How many clusters will compliance isolation force? AWS docs admit shared-cluster isolation cannot match dedicated clusters; regulated environments (finance, gov) often require separate subnets/clusters per security domain. Every extra cluster adds a control plane and redundant nodes, eroding the resource-utilization gains that justify Kubernetes.

Coexistence period: how long, who staffs it, single monitoring entry? Without concrete answers, projects stall at "new platform runs 30% of apps, old platform cannot retire a single VM."

A decision matrix maps each dimension (team capability, app statefulness, compliance isolation, ops headcount) to signals favoring "unified base," "multi-base coexistence," and veto conditions.

Three Migration Strategies and Their Bets

Option A: Status Quo (Multi-base) — Bet: current platform stable; growth doesn't need elasticity. Developer perception: no change. Platform team burden: long-term multi-platform ops. Typical fit: small team, imminent xinchuang (indigenous tech) replacement.

Option B: Keep Experience Layer, Swap Base to K8s — Bet: can extract a stable platform contract (app description, build spec, deploy strategy, config/secret location, observability entry). Developer perception: nearly invisible; must communicate capability gaps (e.g., Beanstalk Cluster Mode only supports rolling and all-at-once deployments; immutable deploys remain Standard-only). Platform team burden: heavy upfront (abstraction + coexistence), then drops. Typical fit: large legacy estate, high stateless ratio.

Option C: Full K8s Native (Helm, Operators, GitOps) — Bet: team has strong cloud-native skills; mostly new microservices. Developer perception: developers learn Deployment, Service, probes, quotas. Platform team burden: cluster ops + developer support. Typical fit: new business lines, greenfield microservices.

Cost & Risk Across Five Dimensions

Migration Cost: Coexistence Often Exceeds Refactoring

Formula:

Migration Effort = (Apps to Refactor × Person-Days/App) + (Coexistence Months × Extra Ops Headcount × Workdays/Month)

.

Illustrative example (hypothetical numbers): 80 apps × 5 person-days = 400 person-days refactoring. Coexistence 12 months × 2 extra ops × 21.75 workdays = 522 person-days. Break-even at ~9.2 months; at 24 months coexistence cost is 2.6× refactoring cost. Many plans only budget the first term.

Resource savings must be calculated against the post-isolation cluster count, not an ideal single cluster. Beanstalk pricing shows Cluster Mode adds fixed per-cluster EKS hourly fees and EKS Auto Mode management fees not offset by Savings Plans/Spot; same principle applies on-prem: every extra cluster adds control-plane overhead and reserved capacity.

Skills: B Concentrates, C Distributes

Option B keeps Kubernetes expertise in the platform team; Option C pushes Deployment, Service, probes, quotas to every dev team — training and error costs scale with team count.

Stateful Workloads: Don't Migrate First

Databases, MQ, file services on StatefulSet/Operators introduce storage failover, backup/restore drills, upgrade data-compatibility — operational complexity far beyond VMs. Leave them on the old base, wired into the unified experience layer.

Xinchuang (Indigenous Tech) Adaptation: Avoid Stacking Two Migrations

Target stacks with Kunpeng/Phytium (ARM), Hygon (x86), Loongson (LoongArch) require multi-arch images and compatibility passes across container runtime, K8s distro, CNI, CSI. Overlapping base migration with middleware indigenization makes root-cause isolation nearly impossible. Sequence them: finalize indigenous stack first, then migrate base once.

Fault Domain: Unified Base Unifies Failure

Ten VM clusters — one failure affects 10%. One shared K8S cluster — control-plane or CNI outage affects 100%. Mitigation: split clusters by business criticality (core systems isolated, non-core shared), but each split reduces the resource-sharing benefit calculated earlier.

Three System Classes That Should Not Migrate Now

Small teams (2-3 platform engineers) — cannot sustain Kubernetes ops; managed services blocked by compliance. Standardize release process instead.

Core stateful systems (ledger, trading, settlement) — elasticity/deployment density gains minimal; risk maximal. Keep on VMs/bare metal, plug into unified experience/observability entry.

Systems facing xinchuang replacement within 1-2 years — don't let base migration and indigenization hit the same system in the same year. Lock target stack first, then migrate once.

Four-Phase Transition Roadmap (with Exit Criteria)

Phase 1: Inventory. Tag all apps (4 buckets), list current bases, baseline coexistence cost. Exit: every app tagged; cost baseline agreed by all stakeholders.

Phase 2: Experience Layer Abstraction. Define platform contract (app descriptor, build spec, deploy strategy, config/secrets, observability). Onboard old PaaS and VM clusters to this contract — zero apps migrated yet, but developer entry unified. Exit: all new apps use contract; legacy contract coverage hits target % agreed at kickoff.

Phase 3: Base Swap. Stateless & new apps onto K8s first; cluster split by business criticality; each batch has rollback window, rollback drill passed before completion. Exit: coexistence duration ≤ cap set at kickoff (referencing the cost model).

Phase 4: Target State. Experience layer unified; base "one primary (K8s) + multiple auxiliaries" — strongly stateful, special runtime, xinchuang-transition systems retain other bases. Not 100% unification.

Every phase must have exit criteria; without them, a transition state becomes a permanent state.

Seven Questions to Bring Into the Review Meeting

Are experience layer and base decoupled? If not, do that first.

What % of inventory is stateless or refactorable?

When K8s control plane / CNI / CSI fails, who in the platform team owns the fix?

How many clusters will compliance force? After splitting, how much resource saving remains?

Planned coexistence length? Where do the dual-platform on-call heads come from?

Any xinchuang replacement planned for these systems in the next 1-2 years?

Are core systems in independent fault domains? Has rollback been drilled?

Experience layer unification is mandatory; base unification is optional. The former decides developer productivity; the latter only pays off when apps are mostly stateless and the platform team can actually operate Kubernetes.