NVIDIA OpenShell: Kernel-Level AI Agent Sandbox with SMT-Verified Policies

One-Line Summary: Turning Permissions into Visible Boundaries

OpenShell is an open-source secure private runtime for autonomous agents (19k+ GitHub stars, Apache 2.0, v0.1.5, written in Rust). Its goal: give agents the ability to read files, install packages, call APIs, and use credentials — but never grant them unlimited access to your data, keys, or network. You write a policy declaring what each agent may touch; OpenShell enforces it at runtime.

Two core mechanisms differentiate it:

Kernel-level enforcement: Each agent runs in an isolated sandbox. Kernel-level controls restrict file access, system calls, and every network connection must pass a policy check before leaving the sandbox.

Formal verification via SMT solver: Before a policy change takes effect, an SMT solver checks whether the relaxation opens new risk paths — e.g., reaching a previously unreachable host with credentials, or calling a new API method. If risk is found, the change is blocked pending human review. This is not mere config validation; it mathematically proves the policy stays within a defined boundary.

Three Roles Define the Trust Boundary

Gateway (control plane): Manages sandbox lifecycles, authentication, policy distribution, credentials, and authorization decisions. CLI, SDK, and TUI interact through it.

Supervisor (trust boundary): Receives agent network requests, evaluates them against policy, opens the real connection, and injects credentials. It runs outside the sandbox.

Sandbox (agent side): Hosts the agent process, identifies which program initiates each request (via trusted /proc observation, not agent self-reporting), and forwards requests to the Supervisor. The sandbox never makes decisions; it only reports intent.

Agent network view: all outbound traffic is cut off except a single mediated channel to the Supervisor. Real API keys live outside the sandbox; the Supervisor injects them on the fly per request.

A Network Request Traverses Five Steps

Agent initiates a TCP connection or DNS query.

Sandbox identifies the requesting program via /proc (immune to command-line spoofing).

Request travels over the OpenShell Sandbox Protocol — a mutually authenticated HTTP/2 connection carrying multiple independent streams (Docker/Podman use Unix sockets, Kubernetes uses TCP, MicroVM uses vsock). Each TCP connection gets its own stream and backpressure, so a slow download won't block DNS or exec.

Supervisor checks policy: can this program reach this host:port? With which credential?

If allowed, Supervisor opens the real connection, injects the credential, and forwards traffic. If denied, the response tells the agent what was blocked and how to request access; OpenShell even auto-drafts a narrow rule proposal that goes through the same verification and review flow.

Policy Authoring and SMT-Based Verification

Policies are YAML declaring allowed filesystem paths, process identities, network destinations, API calls, and credential scopes. Example:

version: 1
filesystem_policy:
  read_only:
    - /usr
    - /etc

To verify a candidate policy against a boundary policy, run:

openshell-prover check candidate.yaml --boundary boundary.yaml

The prover returns one of five results (only within_boundary with exit code 0 passes): within_boundary (0): candidate stays within boundary for checked domains. exceeds_boundary (1): violation found; a counterexample is provided. error (2): check failed to run. unsupported (3): policy contains constructs the prover cannot model (e.g., GraphQL rules, or path comparisons where symlinks could escape). It honestly reports "I cannot check this" rather than pretending success. inconclusive (3): timeout or incomplete run.

Checked domains: filesystem access, process identity, Landlock settings, network connections, REST requests. Coverage does not guarantee the policy is minimal, nor that the running sandbox actually enforces it. A parent agent can also use the prover to ensure child-agent policies don't exceed its own delegated authority.

When Agents Hit Walls, They File Requests

Blocked agents can submit proposals via a local HTTP endpoint ( policy.local) reachable only inside the sandbox. With the policy advisor enabled, the sandbox receives a guide ( /etc/openshell/skills/policy_advisor.md), an agent skill, and enriched denial responses. Proposals specify host, port, binary path, and optionally HTTP method/path.

Two approval modes: manual (default, every proposal requires human sign-off) and auto (proposals passing risk checks auto-approve). Auto-approve is not "allow all"; a hard-coded high-risk destination list forces manual review for:

Link-local addresses ( 169.254.0.0/16, fe80::/10) and cloud metadata hostnames.

Private IP hosts or wildcard hosts. allowed_ips containing private CIDRs or missing host.

Ports above 49152.

Well-known database/cache ports (5432, 6379, etc.).

The prover also flags four risk findings that mandate manual review: link_local_reach: can reach link-local or cloud metadata endpoints. l7_bypass_credentialed: can send unchecked traffic (git, ssh, nc) to a credentialed host. credential_reach_expansion: can use provider credentials on a previously unreachable host:port. capability_expansion: can call new HTTP methods on an already-credentialed host:port.

Rejected proposals return reasons and risk findings, enabling the agent to submit a narrower request — a loop that teaches precise expression, not evasion.

Swap Runtimes Without Changing Policy

Four isolation backends, each establishing boundaries differently:

Docker / Podman: Supervisor runs in a dedicated container; communication via driver-specific Unix socket on a dedicated volume; container network disabled ( network=none).

Kubernetes: Supervisor in its own Pod; Service + mutual TLS; NetworkPolicy allows only Supervisor.

MicroVM: Supervisor is a host process communicating via vsock; the guest has no network device at all .

In every case the sandbox has exactly one egress: the mediated channel to Supervisor. Decision logic is unified, so the same policy behaves identically across runtimes. Supported platforms: Linux (x86_64, arm64) fully; macOS (Apple Silicon + Docker Desktop); Windows (WSL 2, experimental).

First-Run Walkthrough

curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh

Three commands:

openshell profile import --url https://raw.githubusercontent.com/NVIDIA/OpenShell/main/providers/openrouter.yaml
openshell provider create --name openrouter --type openrouter --from-existing
openshell sandbox create --name my-agent --from ghcr.io/anomalyco/opencode:latest --provider openrouter -- opencode -m openrouter/nvidia/nemotron-3.5-lightning:free

Import provider template, store OpenRouter API key in credential store, create sandbox and launch agent (free model used in example). To approve pending rules: openshell rule get my-agent --status pending, then openshell rule approve my-agent --chunk-id <id> or rule approve-all my-agent. Approved rules hot-reload without sandbox restart. Skills package for AI coding tools: npx skills add NVIDIA/OpenShell. SDKs: Python ( uv add openshell), TypeScript, Go, Rust.

Real-World Issues from GitHub

Issue 4098: Chromium Sandbox Conflict

OpenShell's hardening blocks CLONE_NEWUSER, unshare, setns, chroot. Chromium's own sandbox depends on user namespaces; unshare -Ur fails with "Operation not permitted". Workarounds are painful: disable browser sandbox (major security regression) or move browser to a separate VM with CDP routing (fragile). The reporter abandoned OpenShell partly due to this and lack of virtioFS support in MicroVM.

Issue 4133: doctor check False Positives

openshell doctor check

reports "all checks passed" but sandbox fails to start. Root cause: doctor only validates Docker responsiveness on the Mac host, not the Linux VM inside Docker Desktop where the sandbox actually runs. Required preconditions — Landlock ABI 3+ and host networking enabled in the VM — are not checked. One user's Docker Desktop 4.55.0 VM kernel lacked Landlock ( # CONFIG_SECURITY_LANDLOCK is not set), yielding cryptic "fully incompatible access-rights" errors. Community diagnosis: any check reading host kernel or host network stack answers the wrong question.

Issue 2675: Sandbox Cleanup on Failure

Failed sandbox creation leaves residual resources (error state), blocking retries with the same name until manual delete. Discussion highlighted missing provisioning contract distinctions for "in-progress", "terminal failure", and "user-cancelled" states. The repository has 376 open issues, many at this severity level.

Is It Worth Trying?

If you feed real credentials to agents (GitHub tokens, cloud API keys, DB passwords), OpenShell's core value is immediate: agents get "can use this token for this task", not the token value itself.

If you only run local coding agents on your own code , the full stack may be overkill. Docker mode installs easily, but policy authoring has a learning curve.

Isolation strength vs. compatibility tension: Hardening rules break tools relying on namespaces, mounts, or special syscalls (Chromium case is not unique).

doctor check is currently unreliable: It passes even when Landlock or host networking preconditions are missing; you must verify those yourself.

Auto-approve requires caution: It permits arbitrary public-host access for sandboxed binaries if no risk-list item is hit. Docs explicitly state: enable only if you accept that.

Still 0.1.x: Five patch releases (0.1.0–0.1.5), API evolving, Windows not ready, v0.1.0 release notes 30k+ words mostly PR lists.

Ecosystem signal: Built-in MCP server ( https://docs.nvidia.com/openshell/_mcp/server), llms.txt support, AGENTS.md, public skills packages. The project is developed agent-first — "OpenShell is built agent-first" is literal.

Cloud trial available via Brev Launchable (no local setup needed).

Final Assessment

The most transferable insight is the "agent proposes → formal verification → human approves → hot-reload" closed loop. Most agent sandboxes stop at "can it run?"; OpenShell adds "does this relaxation accidentally open other holes?" and answers with an SMT solver that produces counterexamples. That reasoning pattern applies to any "AI operates high-privilege resources" scenario.

Technically it's not vaporware: 19k+ stars, Apache 2.0, ~monthly stable releases, four-language SDKs, Helm charts, Kubernetes deployment path, formal SBOM and signing. NVIDIA treats it as serious infrastructure.

Engineering realities remain: hardening/compatibility tension, SMT verification complexity, 0.1.x API flux. These aren't fatal flaws — they're "too early for production, but direction is right."

If you have agents touching real credentials, try Docker mode once. Cost: ~15 minutes. Benefit: fundamentally changes how you think about "giving AI permissions."

Project: https://github.com/NVIDIA/OpenShell
Docs: https://docs.nvidia.com/openshell/latest/