Running Agents as Kubernetes Workloads: agent-substrate's Multiplexing Runtime

It Operates at a Different Layer Than Vercel Eve or LangChain

Vercel Eve, LangChain, CrewAI, and ADK focus on how to compose agents — wiring models to tools and defining workflows. They assume an agent runs as a long-lived process on a laptop or in a container. Agent Substrate addresses a separate problem: how to run agents as stateful Kubernetes workloads safely and densely.

Agents spend most of their time waiting — for model responses, tool calls, or user input. Running each agent as a dedicated pod wastes resources because the pod idles while holding CPU and memory. Agent Substrate introduces the concepts of actor (the agent instance) and worker (the Kubernetes pod). Actors spend most of their lifecycle suspended, with their full state (memory + filesystem) serialized to GCS or PostgreSQL. When work arrives, the control plane teleports the actor onto any free worker, restoring state in under 400 ms (P95 target 100 ms). After the task finishes, the actor is snapshotted back to storage and the worker returns to the pool.

The article cites a density of 250 actors on 8 worker pods (10×–30× density). Worker pool size is managed by HPA; actor count has no hard upper bound.

End-to-End Request Flow

External request or model API call enters atenet-router, a lightweight proxy that identifies the target actor.

Router queries ate-api-server (control-plane gRPC service) for the actor's current location.

If the actor is idle with state in GCS, the API server assigns a free worker pod.

On the target node, the atelet DaemonSet pulls the actor's snapshot from GCS and injects it into the worker's sandbox.

The worker runs either ateworker-gvisor (gVisor checkpoint/restore) or ateworker-microvm (cloud-hypervisor microVM) to resume the actor as a live process.

During execution, any outbound call to external APIs (OpenAI, Anthropic, GCP) passes through atenet-egress gateway. The actor never holds API keys; credentials are injected at the gateway via MITM interception with a trusted bundle.

When done or timed out, the worker snapshots the actor back to GCS, the actor suspends, and the worker rejoins the pool.

Key security properties: actors cannot communicate with each other (default-deny networking), have no inbound network path except via the router, and credentials never touch the sandbox. Even a successful prompt injection cannot leak secrets.

Complete Architecture: 12 Command Subprojects

The repository is a Go monorepo with 12 independent binaries under cmd/: ate-api-server — core control plane managing actor lifecycle, worker scheduling, snapshots. atelet — node-level DaemonSet coordinating snapshot/restore. atecontroller — Kubernetes controller reconciling WorkerPool custom resources. ate-idp-server — issues OpenID discovery documents and key sets for actor JWT verification. atenet — network suite containing atenet-router (ingress routing) and atenet-egress (credential injection), built on Envoy. ateworker-gvisor — executes gVisor checkpoint/restore inside sandbox pods. ateworker-microvm — cloud-hypervisor variant running actors in microVMs. podcertcontroller — temporary shim signing pod certificates until upstream Kubernetes absorbs the feature. kubectl-ate — CLI plugin adding ate subcommands to kubectl. benchmarking, glutton — synthetic load generators for stress testing.

Additional tooling includes tools/setup-gcp (provisions GKE, GCS, Workload Identity) and three demos under demos/: counter (stateful Go HTTP server demonstrating suspend/resume), sandbox (Alpine Linux shell with persistent filesystem), and claude-code-multiplex (multiple Claude Code agents sharing a worker pool). The architecture documentation explicitly states: "Much of this architecture is aspirational, and is not yet implemented" — parts remain on the roadmap.

Three Key Differentiators

1. Actor Teleportation & State Serialization

Most RAG/agent platforms rely on sticky sessions or external state stores; the agent process itself has no suspend/resume concept. Agent Substrate gives actors real, snapshottable state (RAM + filesystem) that can be moved across workers. This extends Kubernetes container orchestration thinking to the agent layer. The cost is high: GCS/PostgreSQL must handle millions of frequent state updates, and atelet must restore in <400 ms. The north-star goal is 1 billion actors / 1,000 wake events per second — currently aspirational, but the density approach is already noteworthy.

2. Zero Credentials Inside the Sandbox

The atenet-egress gateway runs outside the sandbox on Envoy. It intercepts outbound requests (MITM with trusted bundle) and dynamically injects the actor's credentials into headers. The actor process never sees OpenAI keys, AWS tokens, or GCP credentials. Even if prompt injection forces printenv OPENAI_API_KEY, the output is empty. This eliminates the most painful compliance issue — credential leakage after agent hijacking — at the architectural level, whereas most platforms stuff keys into environment variables.

3. Zero-Trust Networking

Each actor gets its own network namespace. No inbound paths exist except through atenet-router. Lateral traffic between actors is blocked by NetworkPolicy. Control plane ↔ actor communication is also isolated. Identities are bound to SPIFFE X.509 short-lived certificates (5 minutes to 24 hours) tied to the actor and its atespace namespace, enabling verification, audit, and federation to external IdPs. The egress gateway is a pluggable Policy Enforcement Point — DLP, prompt-injection scanners, cloud IAM can be attached without modifying sandbox code.

Installation & Usage

Minimal local path uses a kind cluster (per SOP 5.2):

# Prereqs: Go 1.23+, kubectl, docker
git clone https://github.com/agent-substrate/substrate
cd substrate
# Create kind cluster + local registry
hack/create-kind-cluster.sh
# Deploy ate system + counter demo
hack/install-ate-kind.sh --deploy-ate-system --credential-provider='{"name":"k8s.io"}'
hack/install-ate-kind.sh --deploy-demo-counter
# Install CLI
go install ./cmd/kubectl-ate
# Create an actor
kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter
# Port-forward router
kubectl port-forward -n ate-system svc/atenet-router 8000:80
# Invoke
curl -X POST -H "ate-target-actor: ate-demo-counter/my-counter-1" http://localhost:8000/

The --credential-provider='{"name":"k8s.io"}' flag enables default-deny; actors must be granted access to their atespace namespace's secrets. For GKE, use tools/setup-gcp to provision cluster, GCS bucket, and Workload Identity, then hack/install-ate.sh to deploy (30–60 minutes). Do not use --credential-provider='{"enabled":false}' — it disables the core zero-credential guarantee.

When to Adopt (and When Not To)

Suitable For

Teams building enterprise agent platforms that need to run many stateful agents on Kubernetes.

Organizations wanting zero-trust sandboxing for sensitive data without building isolation layers themselves.

Teams already on Kubernetes with Prometheus/GKE/ArgoCD experience seeking agent scheduling capabilities.

Users of Google Agent Executor (ax) or CNCF kagent who need the underlying runtime.

Not Suitable For

Local development or running a handful of agents — kind + 8 pods + demo is heavy; if it doesn't run locally, don't force it.

Pure stateless agent APIs where sub-100ms cold starts (e.g., AWS Lambda) suffice. Agent Substrate targets workloads requiring session state persistence across invocations.

Projects needing an agent SDK. Substrate does not replace Vercel Eve, ADK, or LangChain; it is the Kubernetes runtime those SDKs run on top of.

Hidden Barriers

Pre-1.0, unstable API. The repo makes no backward-compatibility guarantees. Architecture goals (1B actors, P95 100ms resume, 1K wake/s) are roadmap items.

GKE-centric. tools/setup-gcp treats GKE as first-class; egress credential injection uses Workload Identity. EKS/AKS require custom SPIRE/IAM integration.

Steep learning curve. Requires digesting architecture.md, security.md, glossary.md, egress-credential-injection.md, egress-traffic.md plus Kubernetes network policy fundamentals.

Operational complexity. Just spinning up the kind cluster involves understanding 7 hack scripts' dependencies. Production deployment chains GKE + GCS + IAM + Workload Identity.

Agent-substrate/substrate provides an engineered answer to the emerging problem of running agents as Kubernetes workloads. It complements upper-layer agent frameworks: they decide how agents think ; Agent Substrate decides how agents are securely and densely hosted . Despite being pre-1.0 with roadmap exceeding current reality, it leads the agent-platform Kubernetes runtime race — backed by Google's official blog, CNCF kagent adoption, and 5.5k stars in six months.