Comprehensive Overview of Docker: Architecture, Components, Orchestration, Storage, and Ecosystem

Docker is an open‑source container engine originally released by the PaaS provider DotCloud; its source code is hosted on GitHub, written in Go, and licensed under Apache 2.0. Like shipping containers standardize logistics, Docker standardizes packaging and distribution of container images. See the linked article for a detailed introduction.

Docker uses container isolation and resource‑limiting technologies to package applications and their runtime environments into images. The Docker Registry supports uploading and downloading of these images.

Docker also provides Build, Ship, and Run commands, enabling operators to configure Docker once and then deploy containers repeatedly, achieving the “Build Once Run Anywhere” and “Configure Once Run Anything” principles that have driven the explosion of container technology.

Docker follows a client‑server model with a plug‑in architecture; its backend is loosely coupled, allowing independent modules. For a detailed analysis of Docker’s architecture, see the linked article.

Users interact with Docker via the Docker Client, which communicates with the Docker Daemon. The daemon receives requests, processes them through the Engine, and executes each task as a separate Job.

Docker abstracts the underlying container runtime, improving platform independence. Libcontainer abstracts various container runtimes, evolving into runC, which was contributed to the Open Container Initiative (OCI) as a standard.

Docker’s three major orchestration tools are Compose, Swarm, and Machine. Compose defines and runs multi‑container applications using a single configuration file.

Machine is a host‑management tool that simplifies container installation, upgrades, and network configuration through simple commands.

Swarm is Docker’s native clustering tool that manages multiple Docker hosts; Docker also provides a packaging tool called Dockerfile.

Docker’s networking has historically been complex. For details, see the articles on Docker’s native networking and future networking projects.

Libnetwork, a new Docker networking architecture, merges libcontainer and Docker Engine networking code. It introduces the Container Network Model (CNM) with three core concepts: Network Sandbox, Endpoint, and Network.

Weave created a Networking Plugin technology; mature plugins include Networking Plugin and Volume Plugin.

Weave’s solution consists of a user‑space shell script and a virtual router container. The router container is deployed on each host, linking routes across hosts so Docker tools integrate seamlessly.

Weave creates a virtual network that links containers on multiple hosts, allowing automatic discovery and eliminating the need for manual port mapping or link configuration.

Docker enforces security policies; the Notary filter was introduced to ensure safe image publishing and operation.

Below are Docker topics that are less commonly known.

Containers and Container OS

CoreOS, a popular container‑focused OS, is optimized for Docker with a trimmed kernel. It provides two key tools: etcd for service discovery and data synchronization, and Fleet for maintaining cluster state and ensuring service availability.

VMware offers Photon OS, a container OS that runs on VMs and supports Docker, rkt, and PGC containers. Photon can be paired with the Lightwave authentication tool for better permission management.

Docker Containers and Storage

Docker uses a layered copy‑on‑write (COW) storage model (see the AUFS article). The default COW filesystem does not persist data after a container is removed or restarted.

Volumes solve data‑loss issues, but migrating containers does not move volumes. Flocker addresses this by enabling volume migration alongside containers.

Flocker migration consists of a full migration followed by incremental synchronization, typically performed by taking a snapshot, moving all data, then syncing changes.

Flocker implements a Docker Volume Plugin, abstracting storage differences and providing a standard interface to various back‑ends (AWS EBS, ScaleIO, XtremIO, etc.) for cloud platforms, while local storage support remains less mature.

Flocker’s Storage Driver hides underlying storage differences, allowing containers to be moved between hosts while automatically remapping their volumes.

Rex‑Ray and Dogged are open‑source Docker storage projects that abstract volume management via the Docker storage API and Storage Driver, simplifying container‑level storage provisioning.

EMC, through collaboration with Flocker, has contributed to Docker projects Rex‑Ray and Dogged, gaining influence over storage virtualization standards and the future direction of Docker storage.

As Docker evolves, the demand for container‑level storage interfaces will grow, exemplified by Rex‑Ray and Dogged.

Docker and PaaS

Container‑as‑a‑Service (CaaS) emerged alongside Docker, proposing that infrastructure be delivered as containers. Docker’s packaging and distribution model is expected to become the standard for next‑generation PaaS platforms.

Third‑generation PaaS solutions such as DEIS and Flynn are built on Docker, challenging legacy PaaS offerings.

Flynn is an open‑source PaaS that automatically builds and deploys applications to Docker clusters, borrowing heavily from Heroku’s design.

First‑generation PaaS (e.g., Azure, Heroku) gave way to Cloud Foundry and OpenShift, and now a third wave driven by Docker is underway.

Docker and IaaS Platforms

Major IaaS providers (AWS, Google Compute Engine, Rackspace, etc.) support Docker. Docker’s lightweight and portable nature makes it ideal for hybrid‑cloud deployments, reducing vendor lock‑in.

Container‑as‑a‑Service or Orchestration‑as‑a‑Service offerings such as Tutum enable users to run Docker without worrying about the underlying IaaS.

In June 2014 Rackspace partnered with CoreOS to offer Bare‑Metal as a Service (OnMetal), combining cloud flexibility with high‑performance container virtualization.

Kubernetes, backed by multiple vendors, provides a powerful orchestration layer for Docker workloads, potentially marginalizing OpenStack’s role.

Docker and Kubernetes

Kubernetes originated from Google’s internal Omega project to simplify deployment and management of large‑scale Docker applications.

Kubernetes provides container cluster management, resource scheduling, monitoring, lifecycle management, service discovery, load balancing, and can run on public clouds, private clouds, or bare metal, with broad vendor support.

Docker and OpenStack

Docker integration with OpenStack can follow three main approaches: Docker Driver for Nova, Docker Plugin for Heat, or other custom solutions. The Nova driver treats Docker as a hypervisor but lacks advanced Docker features; the Heat plugin leverages Heat templates to manage Docker resources, offering richer functionality but missing quota and image management.

Docker and DevOps

Docker aligns closely with DevOps principles, allowing developers to focus on code while Docker handles environment consistency across development, testing, and production.

Docker enables versioned Dockerfiles, layered filesystem management, and isolated containers on the same VM, simplifying configuration and reproducibility.

It integrates seamlessly with existing CI/CD tools such as Jenkins, Chef, Puppet, Ansible, SaltStack, Nagios, and OpsWorks.

Emerging DevOps‑as‑a‑Service companies (e.g., CloudMunch, Factor.io, Drone.io) are likely to adopt Docker in their pipelines.

Docker and Microservice Architecture

Microservices require lightweight, independent execution environments; Docker provides precisely that, allowing hundreds of containers to run on a single physical host with rapid start‑stop cycles and easy orchestration.

Docker’s container model delivers the ideal environment for microservices: isolated, fast to provision, and easily orchestrated.

Docker Ecosystem

In the 2014 Linux Foundation North America survey, Docker ranked second only to OpenStack among open‑source cloud projects.

The Docker ecosystem now spans orchestration, container OS, application deployment, networking/SDN, hosting, big data, configuration management, and development tools.

Cloud and infrastructure vendors are joining the Docker ecosystem, contributing to standards and building container‑centric services.

Docker is expected to standardize cloud service delivery, enabling clear boundaries between system builders and users, and facilitating cross‑vendor interoperability.

Docker Hub provides a public marketplace for component and application distribution, with private repositories offering secure, high‑speed, multi‑cloud access for enterprise workloads.

Docker’s Challenger: Rocket

CoreOS, a major Docker contributor, announced its own container engine called Rocket (rkt) in 2014, citing divergent development directions from Docker.

Docker’s roadmap aims to build a complete platform encompassing Machine (system configuration), Swarm (native clustering), and Compose (multi‑container assembly), which will shape the surrounding ecosystem and invite further challenges.

Warm Tip: Please search “ICT_Architect” or “Scan” the QR code below to follow the public account for more content.