Docker in the Real World: Lessons, Pitfalls, and the Quest for a Perfect Container Architecture

Project Environment

We use a standard PHP stack (PHP‑FPM + Nginx) on Alibaba Cloud ECS, with the database also provided by Alibaba Cloud.

Motivation

Although a small startup should avoid unnecessary complexity, we needed Docker to test performance improvements (e.g., HHVM, PHP7) and to isolate problematic services that caused high memory usage and timeouts.

A Small Test

Following Docker’s philosophy, we ran PHP‑FPM and Nginx in separate containers, exposing the PHP‑FPM port to Nginx. Initially we used Docker’s link feature, but it proved unreliable when containers restarted or when multiple instances were required. We switched to Consul for service discovery (see related article).

Code resides on the host and is mounted into containers. We split APIs across several containers, performed stress tests for resource isolation, and observed significant performance gains with HHVM. The architecture is illustrated below.

Logging and Monitoring

We first mounted a host directory for logs, then replaced it with a dedicated Logstash container to collect logs from all services. Monitoring is handled by OneAPM, though other open‑source tools are also possible.

The Perfect Path

To achieve a truly container‑native workflow, we adopted a Docker‑based CI pipeline: each release builds a new image (including code, dependencies, and preprocessing), pushes it to a registry, and each node pulls and restarts the container. The process is shown below.

Facing Reality

While the CI flow is automated, it adds minutes to each deployment because building and pushing images is slower than a simple git pull. Container restarts are also costly; PHP‑FPM handles SIGTERM slowly, making a restart of dozens of containers time‑consuming. Upgrading to Docker 1.10 introduced intermittent port‑connectivity issues that required manual network probes to resolve.

Continuing Exploration

Future plans include using Docker Swarm for simple clustering and evaluating Mesos or Kubernetes for larger scheduling needs. The key takeaway is that no architecture is perfect; the best choice balances project requirements, team expertise, and added complexity.