LiteLLM Poisoned Twice in 6 Months: How Higress Uses Wasm Sandboxing & Immutable Releases to Contain Supply Chain Attacks

Attack Timeline: LiteLLM 1.107.0 (October 21, 2026)

At 19:39 Beijing time, LiteLLM 1.107.0 appeared on PyPI. Within 7 minutes, Docker Hub and GHCR latest and main-stable tags pointed to poisoned images. The project, with ~23M weekly downloads, suffered its second supply chain compromise in 7 months (the first was March 24, 2026, versions 1.82.7/1.82.8).

Four-Step Attack Reconstruction

Based on official LiteLLM report and analyses from Wiz, Chainguard, Endor Labs, and Datadog:

Credential theft: Attackers obtained GitHub CLI OAuth tokens from three developers, likely leaked from logs of an internally deployed, internet-exposed LiteLLM instance.

Workflow manipulation: At 18:08, an attempt to modify the release workflow was blocked by branch protection; 5 seconds later, an Owner-privileged token overrode protection. The workflow was changed to build from a typosquatted repo Berri-AI/litellm (vs. official BerriAI/litellm). A second account approved the production release; a third deleted workflow run logs.

Artifact-only poisoning: Source code in the official Git tag was clean. The malicious change lived only in the build config: pyproject.toml gained data = "cook" in the maturin section, causing three files to be installed into site-packages root: cook.pth: single line import cook. Python executes .pth files on every interpreter startup — no import litellm required; pip, IDE background tasks, any script triggers it. cook.py: unpacks and runs a ~61 KB credential stealer. sitecustomize.py: forks a detached daemon that hourly reads a C2 server address from a GitHub "dead drop" and downloads/executes unsigned Python code.

The malicious source differed from the clean tag by 309 files; reviewing only the tag would miss the attack entirely. Uninstalling litellm does not remove these files.

Mass credential exfiltration: The stealer had 16 modules targeting:

SSH, GPG, TLS private keys, shell history

AWS/Azure/GCP credential files and instance metadata; calls GetCallerIdentity to validate

K8s configs, ServiceAccount tokens, Docker config, GitLab Runner, Jenkins, Vault, Terraform secrets

Browser passwords and cookies

Credentials for 15+ AI dev tools (Claude, Codex, Gemini, Cursor, Copilot, etc.)

Environment variables of all readable Linux processes

A LiteLLM-specific module: if DATABASE_URL exists, connects to the proxy DB and queries LiteLLM_CredentialsTable and LiteLLM_ProxyModelTable — where all upstream model vendor API keys are stored. Data was AES-256-CBC encrypted, RSA-4096 wrapped, disguised as health checks, and POSTed to www.litellm[.]dev/healthcheck.

Floating Tags Amplified Impact

Docker latest, main-latest, main-stable all pointed to the poisoned image. Any pod restart pulling these tags during the 2.5-hour window was compromised, even without explicitly upgrading to 1.107.0. Endor Labs: floating tags and unpinned installs let a bad version hit deployments that never chose it.

PyPI quarantined 1.107.0 after ~1.7 hours; Docker Hub revoked tags after ~2.5 hours. But exfiltrated credentials cannot be recalled. LiteLLM advises: if 1.107.0 ran on an internet-connected host, treat all credentials as compromised and rotate everything.

March vs. October: Different Entry, Same Outcome

March 2026 entry: CI dependency Trivy was poisoned (TeamPCP), used to steal PyPI publish token and push malicious wheels directly. October entry: stolen developer OAuth tokens + hijacked release workflow. Post-entry behavior nearly identical:

Install-time execution: Both used .pth startup hooks — no import needed.

Same privileges as proxy process: Malicious code runs inside the LiteLLM process, inheriting its file reads, env vars, network, and subprocess capabilities.

Straight for the "keychain": March payload deployed privileged pods per K8s node; October payload included a dedicated module to query the LiteLLM DB for upstream API keys.

Wiz observed attackers calling bedrock:ListFoundationModels within 5 hours using stolen AWS creds — confirming the goal is model invocation rights.

Why LLM Proxies Are High-Value Targets

Centralized keys: All vendor API keys aggregated in the proxy for unified access and metering.

Privileged position: Runs in production clusters with cloud instance roles, K8s ServiceAccounts, and mandatory public internet access to model APIs.

Extensions = in-process code: In Python proxies, third-party deps and custom callbacks run as part of the proxy process with full privileges.

Key concentration is good (unified audit, rotation), but the third property makes blast radius equal to process privileges. One poisoned dependency exposes the entire keychain.

Higress's Answer: Wasm Sandbox for Extensions

Higress data plane is Envoy (C++). Core gateway functions (routing, LB, TLS) are native Envoy; AI proxy, auth, rate limiting, content safety, MCP, etc. run as Wasm plugins inside Envoy's embedded Wasm VM. Official plugins ( ai-proxy, ai-token-ratelimit, ai-security-guard, etc.) all use this model.

Fundamental difference: Python deps default to full process privileges; Wasm plugins default to only explicitly granted host capabilities. Every plugin action must go through Proxy-Wasm host interfaces: read/write request/response, read own config, read/write shared key-value data, call pre-registered services, log/metrics. Nothing else.

Verifiable in Envoy's proxy-wasm-cpp-host source (

https://github.com/proxy-wasm/proxy-wasm-cpp-host/blob/main/src/exports.cc

):

No auto-execution: No .pth, sitecustomize hooks. Plugins load only when explicitly referenced in a WasmPlugin resource; code runs only in request-handling and config-delivery callbacks.

No filesystem: path_open always returns NOENT; fd_read comment: "does not support reading any file"; fd_readdir: "no real filesystem in proxy-wasm". ~/.aws/credentials, ~/.ssh/id_rsa, /var/run/secrets/kubernetes.io/serviceaccount/token — unreadable.

Env vars invisible by default: environ_get returns only variables explicitly configured in WasmPlugin vmConfig.env. Host process and other processes' env vars do not exist for the plugin.

No arbitrary network: Host does not implement sockets; sock_send / sock_recv return ENOTSUP. External calls must go through Envoy, which checks if the target cluster exists. No registered cluster = no route. Exfiltration to www.litellm[.]dev or metadata endpoint 169.254.169.254 blocked unless explicitly registered as an upstream.

No processes: No fork/exec, no daemons, no systemd, no privileged pod deployment.

Memory isolation: Each plugin accesses only its own linear memory. Cannot read Envoy's TLS keys, other plugins' memory/config. Plugin crash does not crash Envoy; FAIL_CLOSE / FAIL_OPEN policies decide request fate.

Running the 1.107.0 malicious actions through this sandbox: every step fails (see article's Figure 4).

Sandbox Is Not a Silver Bullet

Sandbox limits plugin escape , not in-scope abuse. A plugin still sees requests/responses it handles (could exfiltrate via response), reads its own config, and accesses shared key-value store. ai-proxy must have upstream API keys to forward requests.

Therefore, defense-in-depth:

Trusted plugins only: Traceable source, auditable code (addressed by release pipeline, §5).

Least privilege: Split capabilities across plugins; each gets only needed credentials (auth plugin gets consumer creds, model proxy gets model keys).

Version pinning: WasmPlugin supports sha256 field; mismatch rejects load. Production should use "version + sha256", never latest.

Credential offloading: Enterprise edition uses KMS-encrypted key storage; gateway config holds only encrypted refs (§7).

Immutable Release Pipeline: Locking Every Link LiteLLM Lost

Since v2.2.4 (Aug 2026), Higress official Go/Rust Wasm plugins use a "build once, review snapshot, promote by digest" immutable flow. Process and workflows public at

https://github.com/higress-group/higress/blob/main/docs/developers/immutable-plugin-releases.md

.

Mapping LiteLLM failure points to Higress controls:

Build source: LiteLLM workflow redirected to fake repo. Higress release-prep validates target is exactly current main commit; each candidate artifact records source commit and input hash, summarized in a snapshot file requiring PR review before merge.

Source vs. artifact: LiteLLM tag clean, wheel poisoned. Higress plugin images carry OCI annotations ( org.opencontainers.image.revision, io.higress.plugin.input-hash) binding source commit and input hash. Formal release copies only the already-reviewed candidate artifacts by digest — never rebuilds .

Release approval: LiteLLM approval bypassed with stolen account. Higress credentials for writing public version tags live only in protected plugin-release-production environment; formal version tags require manual release-manager approval, and only a dedicated release App can create them.

Version tags: LiteLLM latest / main-stable mutated to poisoned image. Higress public tags are immutable: publish fails if same tag's digest differs. latest advances only after full batch passes and version monotonically increases. Console, gateway, and Standalone default configs pin exact versions, never latest.

Audit trail: LiteLLM workflow logs deleted. Higress release snapshots and evidence files (with SHA-256) committed to repo. Example:

https://github.com/higress-group/higress/blob/main/plugins/release/snapshots/2.2.4.json

locks 43 official plugins, each with version, image digest, input hash, source commit — anyone can verify.

{
  "logicalId": "ai-proxy",
  "version": "2.0.1",
  "ociRef": "higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/ai-proxy:2.0.1",
  "digest": "sha256:754c4d6c9e8aed2cabc4aa37c21f1e6dbf3b4973f01c18765ab83c17ad4d40db",
  "inputHash": "sha256:fc42f8a92ee198eacb59689be844cbeb00dab79b359b583cb404f4106fef5cee",
  "sourceCommit": "7774663e4353652f5c3cecd7e98a1a43b05ee15c"
}

Images hosted on Alibaba Cloud Container Registry (ACR) with security scanning (detailed in March article).

No process stops all credential theft. Immutable tags and digest pinning mean: even if breached, your running version won't be silently swapped; tampering is detectable.

Actionable Guidance

If you use LiteLLM

Do not start Python/pip on affected hosts — each interpreter start re-runs the stealer.

Use non-Python tools ( find, sha256sum) to check site-packages for cook.pth, cook.py, sitecustomize.py, /tmp/.pc_once_*, /tmp/remote_code_loader.lock; kill the daemon holding the lock.

From a clean machine, rotate ALL credentials: proxy DB model keys, master/virtual keys, DB password, cloud creds/instance roles, K8s tokens, Git/npm/PyPI tokens, SSH keys, AI tool sessions.

Pin to litellm==1.106.2; reference images by digest.

If evaluating AI gateways

Higress covers core LiteLLM proxy scenarios: unified OpenAI-compatible access to major vendors and self-hosted (vLLM/Ollama) via ai-proxy; consumer auth ( key-auth, jwt-auth); token-level rate limiting ( ai-token-ratelimit); multi-model LB/fallback; observability & token accounting ( ai-statistics); content safety ( ai-security-guard); MCP Server hosting. Managed option: Aliyun AI Gateway Serverless Standard (free tier, §7).

If already on Higress

Pin plugin versions with sha256; avoid latest. vmConfig.env inject only vars the plugin truly needs; avoid HOST passthrough of host env vars.

Keep auth plugins on FAIL_CLOSE (CRD comments explicitly discourage fail-open for auth).

Register only business-required upstreams — this list doubles as plugin egress allowlist.

Treat custom plugins like official ones: auditable source, independent build, digest-based release.

Higress Enterprise (Aliyun AI Gateway): Three Extra Layers Beyond Sandbox

Open source Higress gives a secure architecture; self-hosted security still depends on ops discipline: image source, version pinning, credential rotation cadence, vulnerability response speed. LiteLLM's advisory noted: customers using private enterprise registries were unaffected. Managed/enterprise distribution shifts that discipline to a dedicated team.

Aliyun AI Gateway is Higress enterprise edition, same codebase, compatible migration path.

Credentials never land in plaintext: API keys encrypted in KMS; gateway config holds only encrypted refs. No plaintext in config files, env vars, or logs. Full RAM permission model; each consumer has isolated key — blast radius limited.

Supply chain hosted: Gateway engine built, released, upgraded by platform. No pulling from public registries; no watching every update. Custom plugins developed in WebIDE, built in CloudEffect VPC build clusters — code fetch, dependency download, artifact production never touch public internet.

Traffic & content protection: One-click Aliyun WAF integration (SQLi, XSS, command injection, CC, bad bots); AI safety fence detects prompt injection and sensitive content.

Custom plugins still run in Wasm sandbox; platform adds lightweight isolation, strict permissions, code verification.

Performance: Sandbox for Untrusted Code, Native Speed for Core Paths

Isolation has cost. Open source Higress Wasm plugins copy full request bodies into plugin linear memory; multiple plugins processing same large payload multiply copy/allocation overhead. Wasm linear memory grows but cannot shrink, so sustained high concurrency keeps memory high.

Enterprise edition splits concerns: user/third-party extensions stay in Wasm sandbox; platform-owned core AI paths (protocol translation, large body parsing) use Envoy native plugins with AI-specific optimizations — reducing duplicate copies and unnecessary memory pressure. This code maintained in controlled dev/release pipeline.

Aliyun AI Gateway team benchmarked Serverless Enterprise vs. self-hosted open source Higress (same load generator, synthetic JSON, same mock zero-latency backend; different deployment resources/entrypoints, reflecting each deployment's own behavior):

1 MB request: 10–100 concurrency, enterprise avg gateway latency 13–18 ms vs. open source 68.8%–97.1% higher. Max gap at 90 concurrency: 446.2 ms → 12.7 ms.

10 MB request: 82.7%–90.2% reduction. Max at 30 concurrency: 1,688.1 ms → 166.1 ms. At 100 concurrency: enterprise 384.3 ms, open source 2,527.4 ms (~2.14 s saved per request).

Consumer scale another differentiator. Higress July Serverless Enterprise test: at 10k consumers, enterprise auth p50 latency 1.47 ms vs. open source 50.32 ms (~34×). At 20k consumers, enterprise 100% success rate, open source 77.14%. Enterprise separates consumer data from auth config, so per-auth cost doesn't grow linearly with total consumers.

Cost: Pay-As-You-Go From Zero

Serverless Standard: Trial, dev/test, low traffic. Free tier, no instance fee, pay only actual usage (requests + public egress).

Serverless Enterprise: Production workloads; dedicated ingress, custom domain, TLS, 99.95% SLA. Instance fee ~¥0.245/hr (<¥6/day); requests, data processing, egress billed separately.

Dedicated Instance: Higher isolation, performance specs, advanced governance. Billed by instance spec.

(Prices reference; check Aliyun official site.)

"Free tier" means Standard charges no instance fee — gateway itself not billed by time, only by usage. Small traffic = small bill. Upgrade to Enterprise when volume demands dedicated ingress and SLA.

Self-hosted open source costs: compute, network, redundancy capacity for peaks; plus ongoing labor for image maintenance, plugin compatibility checks, upgrade regression testing, incident debugging. Higress July estimate: Serverless reduces cost 70%–90% for small/medium scenarios.

Conclusion

Supply chain attacks won't stop. PyPI, npm, Docker Hub, GitHub Actions — every link can be the next entry point.

In "install-and-run" ecosystems, security relies on luck: you pray every dependency, every maintainer account, every CI pipeline stays clean. In "default-deny" sandboxes, security relies on architecture: even if poison gets in, it stays in the cage.

Open source Higress puts extension code in a Wasm sandbox and locks the release chain into immutability. Higress Enterprise adds managed credentials, hosted supply chain, and integrated WAF/AI safety. Choose based on where your team wants to spend energy. To try managed form first, Serverless Standard starts free — no instance fee upfront.