Why Coinbase Skips Kubernetes: Insights from Their Container Orchestration Journey

1 History

Before discussing the current state, it is useful to understand the evolution of container technology up to today.

1980s: chroot

1990s: jail

Early 2000s: jail > FreeBSD

Mid‑2000s: cgroups

Late 2000s: LXC (Linux containers)

Early 2010s: Docker

Late 2010s: Kubernetes

For a deeper dive, see Enterprise Docker Chapter 7.

Ten years ago, before Docker and other mainstream container runtimes existed, large companies built internal systems to package source code for production. Engineers ran custom, often monolithic builds on their machines, which differed greatly from production environments.

These internal packaging and deployment processes required large operations teams, typically part of a platform or infrastructure organization focused on operational work such as host troubleshooting, OS patching, capacity planning, server provisioning, and post‑deployment tasks, with little automated orchestration.

Some teams could construct a “golden image” using tools like HashiCorp Packer, driven by CI systems such as Hudson (pre‑Jenkins), and distribute these images manually or via configuration‑management tools.

2 Why Do Enterprises Containerize?

Enterprises adopt containerization to boost engineering output and developer productivity in a fast, safe, and reliable manner. Containers enable consistent local development, testing, and production environments, explicit dependency binding, fine‑grained resource definitions, and horizontal scaling.

3 Virtualization

Virtualization allows multiple containers to run on a single OS instance, with each container seeing only its authorized devices/resources. On hosted platforms like AWS, containers run under a hypervisor that manages the underlying VMs.

Virtualization makes today’s container world possible; without it, running multiple applications on shared hardware would be infeasible.

4 Problems Container Orchestration Platforms Must Solve

Orchestration platforms address three main problem classes:

Standardized deployment tools (deployment)

Heuristic‑driven horizontal scaling

Self‑healing via rescheduling/moving containers on failure

Additional features such as storage orchestration, secret/config management, and automatic bin‑packing often require significant investment to scale.

5 What Does Coinbase Use for Orchestration?

Coinbase runs Odin + AWS ASG (Auto Scaling Group). Clicking Deploy in the internal UI (Codeflow) triggers Odin, which starts a Step Function, launches a new VM in an ASG, pulls software from internal sources, performs health checks via a load balancer, and finally switches traffic to the new instances using a blue/green deployment.

Odin implements key Kubernetes features: Deploy/Rollback button, heuristic‑based scaling (custom AWS metrics or CPU), and automatic rescheduling of unhealthy VMs.

A dynamic configuration service backed by DynamoDB provides low‑latency (≈6 ms) access to secrets and configuration for internal clients.

Service discovery and load balancing rely on Route 53, ALB, and gRPC client‑side load balancing (native or via Envoy).

6 Why Not Kubernetes?

Full‑time compute team required – Operating Kubernetes demands a dedicated team to manage dozens of clusters, build tooling, and maintain control‑plane components.

Kubernetes security is complex – Replicating Kubernetes’ security guarantees requires substantial investment from both compute and security teams.

Managed Kubernetes is immature – Early‑stage services like AWS EKS and Google GKE still face operational challenges and limited visibility.

Cluster upgrades and management overhead – Upgrading clusters, patching, and handling rollbacks involve extra steps, often requiring auxiliary clusters for failover.

Current burden is low – Coinbase runs on a hardened OS with minimal dependencies; rollbacks are as simple as changing a line of code, consuming less than five hours per month on related tasks.

7 Kubernetes Security

Securing a Kubernetes cluster that holds over $8 billion in crypto assets involves protecting components such as etcd, kubelet, the API server, and any overlay (e.g., Istio). This requires deep knowledge of namespaces, seccomp, SELinux, cgroups, and adherence to CIS benchmarks.

Relevant CVEs include CVE‑2019‑5736 (runc root escalation), CVE‑2019‑11246 (malicious tar execution), and CVE‑2019‑11253 (API server resource exhaustion).

8 When Might Coinbase Adopt Kubernetes?

If a higher‑level orchestration platform offers compelling use cases that integrate easily with existing tooling, Coinbase will evaluate it. Otherwise, the team will first explore AWS managed services like Fargate and ECS before considering Kubernetes.

Kubernetes would be adopted only if it provides clear benefits—new features, debt reduction, or critical customer requirements—that outweigh current operational costs.

9 What’s the Future of Container Orchestration?

Coinbase may migrate some workloads to higher‑level abstractions such as Fargate or ECS to improve utilization and cost efficiency. The goal is a seamless transition where tooling remains unchanged.

10 Does Coinbase Hate Kubernetes?

No. Despite challenges, Kubernetes is an impressive tool that has advanced the industry. Ongoing developments like Knative, Fargate, and Cloud Run aim to raise the abstraction level and mitigate operational burdens.