How Docker Transformed Development: From VMs to Lightweight Containers

1. Background of Docker

In typical development and project scenarios, developers face issues such as personal development environments with multiple virtual machines for CDH clusters, company internal development environments with shared VMs, and inconsistencies between development, testing, and production environments, leading to cumbersome upgrades and migrations.

These problems stem from the lack of a technology that can shield OS differences while maintaining performance, which Docker addresses.

2. What is Docker

Docker is an application container engine. It uses Linux Namespace for process isolation and CGroup for resource control. Combining these creates a container, a user‑space isolated object with limited resources.

Docker builds on Linux Container (LXC) technology and adds features such as Chroot, Veth, UnionFS, Iptables/netfilter, TC, Quota, and Setrlimit to provide comprehensive resource control.

Docker requires a Linux kernel version of at least 3.8; the official recommendation is kernel 3.10 or higher.

3. Difference from Traditional Virtualization

Traditional virtualization adds a Hypervisor layer between the VM and hardware, which incurs performance overhead and slower startup. Docker runs directly on the host OS using kernel features, resulting in near‑bare‑metal performance and fast container startup.

On Windows, Docker actually runs a lightweight Linux VM via Hyper‑V, so containers still rely on a Linux kernel.

4. Basic Docker Concepts

Engine : creates and manages containers, pulls/pushes images.

Image : a template consisting of a base OS layer and application layers.

Container : a runtime instance of an image with its own namespace and cgroup.

Repository : storage for images, e.g., Docker Hub, Alibaba Cloud, Tencent Cloud.

Host : the server where the Docker engine runs.

5. Analogy with VMs, Git, and JVM

Docker images are similar to VM images, but containers share the host kernel, eliminating the heavy VM overhead. Docker’s repository model mirrors Git’s versioning. Docker’s slogan “Build, Ship, and Run Any App, Anywhere” parallels Java’s “Write Once, Run Anywhere”.

If an application uses kernel features newer than the host’s kernel, it may fail unless the host kernel is upgraded.

6. Docker Image Filesystem

Docker images use a layered storage format. Each layer can be shared among multiple images. UnionFS (often AUFS) merges these layers into a single view. Writable layers are created under /var/lib/docker/aufs/diff and mounted at /var/lib/docker/aufs/mnt when a container starts.

UnionFS’s copy‑on‑write allows containers to start in seconds and run thousands on a single server.

7. Base Operating Systems for Docker

BusyBox : ultra‑minimal Linux (~2 MB), suitable for simple tests.

Alpine : lightweight, security‑focused (~5 MB), widely used in production.

Debian/Ubuntu : full‑featured (~170 MB), ideal for development.

CentOS/Fedora : enterprise‑grade (~200 MB), suited for production.

8. Persistent Storage

Since a container’s writable layer disappears on stop, Docker offers two persistence methods:

Bind‑mount a host directory into the container.

Use network‑shared storage (e.g., NFS) and mount it into the container.

9. Creating Docker Images

Two methods:

Commit a running container to a new image.

Write a Dockerfile and build an image from it.

FROM ubuntu/14.04
MAINTAINER guest
RUN apt-get install openssh-server -y
RUN mkdir /var/run/sshd
RUN useradd -s /bin/bash -m -d /home/guest guest
RUN echo 'guest:123456'| chpasswd
ENV RUNNABLE_USER_DIR /home/guest
EXPOSE 22
CMD ["/usr/sbin/sshd -D"]

10. Docker Use Cases

Lightweight VMs for testing.

Cloud hosts via Kubernetes or other orchestration platforms.

Packaging application services (e.g., Java + Tomcat).

Container‑as‑a‑Service (CaaS) compared with IaaS, PaaS, SaaS.

CI/CD pipelines for automated build, test, and deployment.

Microservice isolation to avoid OS compatibility issues.

Running temporary one‑off tasks.

Multi‑tenant environments with isolated containers.

11. Conclusion

Docker is not a mysterious technology; it integrates mature Linux kernel features to provide application‑level containerization, enabling “build once, run anywhere” for Linux‑based systems. As container adoption grows, orchestration tools like Kubernetes, Mesos, or Swarm become essential for managing large‑scale deployments.