Cloudflare Fixes Cross-Tenant Data Leak in Containers via Storage Allocator Flaw

Cloudflare disclosed a cross-tenant data leak vulnerability in its container platform (also used by Cloudflare Sandboxes). Customers with Workers Paid accounts could recover disk blocks left by deleted containers on the same host, exposing databases, credential files, and other sensitive data.

Root Cause: Storage Allocator Configuration

The flaw was not in the Firecracker VM boundary but in the underlying storage allocator. Each container runs in a Firecracker VM with a writable root disk provisioned via Linux device-mapper thin provisioning. The affected storage pool used a 64 KB thin block size and the skip_block_zeroing parameter, which tells dm-thin not to zero newly allocated blocks before handing them out.

Writing a single 4 KB block to an unmapped region causes dm-thin to allocate a 64 KB physical block from the shared pool. The write only overwrites the 4 KB written; up to 60 KB of the block retains data from the previous container. Direct disk reads can therefore return bytes the new container never wrote.

Researcher Discovery and Scope

Accomplish security researcher Oren Yomtov reported the issue via Cloudflare's bug bounty on September 4. Researchers used ext4 directory block checksums to distinguish their own directory blocks from others. Across six production environments, all 5,614 testable directory blocks originated from other tenants, revealing 2,700 distinct foreign inodes. Residual data appeared in 18 of 24 deployments and on 20 of 22 underlying nodes spread across four continents. Recovered blocks included directory structures, database pages, and intact SQLite databases.

Cloudflare's Scoping and Limitations

Cloudflare clearly bounded the issue: attackers cannot choose targets, workloads, or hosts; cannot read currently attached disks; and cannot guarantee residual data exists. Exposure depends on where Cloudflare places workloads and which freed blocks dm-thin reallocates. No evidence showed attackers modified other customers' data or affected availability.

Expert Commentary: Storage-Layer Isolation

Visa senior cloud security engineer Peter Ward described this as a recurring pattern: "This is tenant isolation broken at the storage layer, not an application-level bug." He argued that ephemeral disks and shared hosts need explicit zero-on-allocate or wipe-on-free checks in the control plane, not just network isolation. Ward offered a test for any container or serverless platform: ask whether every reused volume path is zeroed on allocation, and treat residual data on shared storage as a first-class isolation control.

Cloud and network security lead Sherin Shahanas noted root causes are rarely proven: "'Isolation' is often an architectural claim, not an independently tested property at the storage layer." He challenged teams running shared infrastructure (containers, VMs, or shared virtualized clusters) to verify whether freed blocks are zeroed before reuse.

Industry Context

The disclosure coincided with Microsoft's launch of Azure Container Apps Sandbox (hardware-isolated micro-VM boundaries) and Google's GKE Pod Snapshots benchmark (gVisor checkpointing memory and disk to Cloud Storage). Both vendors are selling isolation for untrusted agent workloads.

Lesson: Verify Below the Announced Layer

Cloudflare's disclosure reminds operators that security guarantees must cover all layers beneath the one advertised — including the block allocator. The Firecracker boundary held; the leak existed beneath it, in a performance option that skipped module-reuse zeroing.

Remediation Timeline and Actions

Sep 4 15:26 UTC: Report received.

18:45: Incident created.

21:27: Runtime fix merged.

22:03: Changes for new and existing pools merged.

23:15: Deployment started; completed Sep 7.

Cloudflare removed the skip_block_zeroing config, restoring default zeroing of newly allocated blocks. However, this did not scrub blocks already mapped into existing thin devices — whether in running container disks or in per-host snapshot caches for OCI image layers. New containers could inherit those mappings without reallocation. Therefore, Cloudflare drained all running container disks, evicted hosts off-peak, rebooted VMs, and cleared every host's image cache. All work finished by September 19.

Verification and Aftermath

Cloudflare built a signature based on the write-small-read-large pattern and scanned retained historical disk I/O telemetry. Only activity attributable to the researchers and Cloudflare's own engineers during authorized validation was found. Researchers confirmed their proof-of-concept stopped working after the changes, and recovered data under their control was securely deleted. Cloudflare paid the bug bounty on September 14.

Cloudflare did not disclose log retention duration or when the unsafe setting was first deployed, so the total exposure window remains unknown. Cloudflare Sandboxes — marketed as a secure environment for untrusted code including AI agents — were also affected. The impact extends beyond individual accounts: multi-tenant Kubernetes clusters, CI runners, and shared SaaS infrastructure that rely on server-level isolation face the same risk when the underlying filesystem uses thin provisioning with wiping disabled.

The incident shows that Linux kernel issues can propagate widely in infrastructure, and overlooked configuration options amplify that risk.