How Dragonfly’s P2P Architecture Earned CNCF Incubation

Background

On 10 April 2023 the Cloud Native Computing Foundation (CNCF) Technical Oversight Committee promoted the open‑source project Dragonfly to the incubation stage, making it the third Chinese project in CNCF after Harbor and TiKV.

Architecture

PouchContainer – a lightweight container engine that initiates image pull requests.

Registry – stores container images as a set of layered files.

SuperNode – the server‑side component that creates seed blocks, builds the peer‑to‑peer (P2P) network and schedules block transfers.

Block – a seed block generated from an image layer; blocks are distributed among peers.

DFget – the client daemon installed on each host; it intercepts pull commands, uploads/downloads blocks and talks to the container daemon.

Peer – any host that participates in downloading the same file from other peers.

Image Pull Workflow

PouchContainer issues a docker pull (or equivalent) command; DFget intercepts the request.

DFget sends a scheduling request to a SuperNode.

The SuperNode checks its local cache. If the requested layer is absent, it fetches the layer from the Registry, splits it into seed blocks and stores them; if cached, it directly creates block‑download tasks.

DFget receives the task list and downloads blocks either from the SuperNode or from other Peers. When all blocks of a layer are received, the layer is handed to the container engine. After all layers are assembled, the full image is ready.

Key Features

Large‑Scale Image Download

Dragonfly uses a P2P model so that each additional Peer contributes upload bandwidth. The system therefore scales out: more peers result in higher aggregate download throughput, eliminating the single‑point bandwidth bottleneck of traditional registries.

Long‑Distance Transfer

SuperNode instances are deployed in a CDN‑like hierarchy. The first requester creates seed blocks that are cached locally; subsequent peers download those blocks from the nearest SuperNode, avoiding cross‑region traffic. Dragonfly also implements an automatic hierarchical pre‑warming mechanism: during an image push, each layer is immediately replicated to nearby SuperNodes, so future pulls can be satisfied from the cache without remote fetches.

Bandwidth Cost Reduction

Dragonfly applies dynamic compression only to high‑value blocks when both SuperNode and Peer have sufficient CPU/memory resources. The compression is multi‑threaded and performed once per block, yielding up to ten‑fold bandwidth savings compared with native HTTP compression. Additionally, the scheduler prefers intra‑datacenter Peer‑to‑Peer transfers, further reducing cross‑network traffic.

Secure Transmission

Supports HTTP header authentication for protected registries.

Uses a custom data‑storage protocol that can be wrapped with optional encryption modules.

Pluggable encryption plugins are planned for future releases.

Multi‑stage verification (checksum and signature checks) prevents tampering of transferred blocks.

Promotion to Incubation

Dragonfly satisfied CNCF incubation criteria: mature codebase, broad adoption in production (Alibaba Cloud, China Mobile, Shopee, Bilibili, Ant Financial, etc.), a diverse contributor base, and a clear solution to large‑scale image distribution that differs from centralized registries. The TOC vote reflected the project's technical merit and its potential to enrich the cloud‑native ecosystem.

Future Work and Ecosystem Integration

As an incubating project, Dragonfly will focus on:

Improving installation, upgrade and configuration workflows to lower the entry barrier for enterprise users.

Enhancing security hardening and exposing stable encryption plugins.

Deepening integration with other CNCF projects such as Harbor, Quay and Clair to provide end‑to‑end image lifecycle management.

Contributing to OCI Distribution standardization efforts.