Avoid These 10 Common Docker Pitfalls in Production

Background

Docker is a core infrastructure component for modern operations and development, but many issues that do not appear on physical or virtual machines surface in container environments. This guide extracts ten common production‑grade Docker pitfalls from real incidents, providing symptom descriptions, root‑cause explanations, diagnostic commands, fixes, and preventive actions.

Pitfall 1: Disk Full

Symptoms

Container fails to start with

no space left on device

docker ps

reports

Cannot connect to the Docker daemon

df -h

shows /var/lib/docker at 100% usage

Writes to files return "No space left on device"

Root Cause

The Docker storage driver (overlay2, devicemapper, btrfs, zfs) stores images, containers, logs, and build cache under /var/lib/docker. If this directory resides on a partition without a dedicated mount or on a root partition with limited space, logs and caches quickly fill it.

Diagnostic Commands

# Check disk usage of Docker data directory
df -h /var/lib/docker

# Show Docker disk usage breakdown
docker system df

# Detailed usage per component
docker system df -v

# Inspect container log sizes
ls -lh /var/lib/docker/containers/*/*-json.log

# Check overlay2 layer usage
du -sh /var/lib/docker/overlay2/*

Fixes

# Clean up dangling images
docker image prune -a

# Remove build cache
docker builder prune -a

# Remove all unused resources (images, containers, networks, caches)
docker system prune -a --volumes

# Limit container log size (daemon config or docker‑compose)
# Global limit via /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {"max-size": "100m", "max-file": "3"}
}

# Manual log truncation (emergency)
> /var/lib/docker/containers/<container-id>/*-json.log

Preventive Measures

Place /var/lib/docker on a dedicated partition or LVM volume.

Configure log rotation (max‑size + max‑file).

Regularly clean unused images and caches.

Monitor disk usage and alert above 80%.

Pitfall 2: Time Drift Inside Containers

Symptoms

date

inside container differs from host by 8 hours.

Application log timestamps are incorrect.

Database writes have an 8‑hour offset.

Certificate validity calculations are wrong.

Root Cause

Containers inherit the host kernel and lack a separate timezone setting. If the host runs in CST (UTC+8) but the container does not mount the timezone files, it defaults to UTC.

Diagnostic Commands

# Host time
date

# Container time
docker exec <container-id> date

# Check if timezone files are mounted
docker inspect <container-id> | grep -A 20 "Mounts"

Fixes

Option 1: Mount timezone files at runtime

docker run -v /etc/timezone:/etc/timezone:ro \
           -v /etc/localtime:/etc/localtime:ro \
           nginx

Option 2: Set TZ environment variable (supported by some base images)

docker run -e TZ=Asia/Shanghai nginx

Option 3: docker‑compose configuration

services:
  app:
    image: my-app:latest
    environment:
      TZ: "Asia/Shanghai"
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

Option 4: Set timezone in Dockerfile

FROM ubuntu:20.04
RUN apt-get update && apt-get install -y tzdata && \
    ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo "Asia/Shanghai" > /etc/timezone

Pitfall 3: DNS Resolution Failure for Internal Names

Symptoms

Host can ping redis-master, container cannot. curl http://nginx works on host but fails inside container.

Container resolves public DNS (e.g., baidu.com) but not internal domains.

Cross‑container communication errors like "could not resolve host".

Root Cause

Docker’s built‑in DNS server (127.0.0.11) knows names created via --link or docker network, but it does not automatically forward queries to the host’s custom DNS servers (e.g., corporate DNS entries).

Diagnostic Commands

# View container DNS configuration
docker exec <container-id> cat /etc/resolv.conf

# Inspect container network mode
docker inspect <container-id> | grep -A 10 "NetworkSettings"

# Test DNS inside container
docker exec <container-id> nslookup nginx
docker exec <container-id> dig nginx

# Host DNS configuration
cat /etc/resolv.conf

Fixes

Option 1: Use --dns to specify DNS servers

docker run --dns 192.168.1.53 nginx

Option 2: Configure DNS in docker‑compose

services:
  app:
    image: my-app:latest
    dns:
      - 192.168.1.53
      - 8.8.8.8
    networks:
      - my-net

networks:
  my-net:
    driver: bridge
    ipam:
      config:
        - subnet: 172.20.0.0/16

Option 3: Global daemon DNS configuration

{
  "dns": ["192.168.1.53", "8.8.8.8"]
}

Note: After editing daemon.json , run systemctl restart docker for changes to take effect.

Pitfall 4: OOMKilled Processes

Symptoms

docker ps

shows container exited.

Last log line appears normal; no error. docker inspect shows OOMKilled: true.

Host dmesg or journalctl contains OOM messages.

Root Cause

Memory limits are enforced by Linux cgroups. When a process requests more memory than the container’s limit, the OOM killer terminates a process. If the process does not handle SIGKILL, the container exits abruptly.

Diagnostic Commands

# Check container exit status
docker inspect <container-id> | grep -E "OOMKilled|ExitCode|State"

# View container memory usage peak
docker stats <container-id> --no-stream

# Inspect memory limits
docker inspect <container-id> | grep -A 5 "Memory"

# Host cgroup memory stats
cat /sys/fs/cgroup/memory/docker/<container-id>/memory.usage_in_bytes
cat /sys/fs/cgroup/memory/docker/<container-id>/memory.limit_in_bytes

# Host OOM logs
dmesg | grep -i "out of memory"
dmesg | grep -i "killed process"
journalctl | grep -i oom | tail -20

Fixes

# Emergency: increase memory limit and restart container
docker run --memory=1g my-app:latest

# docker‑compose example
services:
  app:
    image: my-app:latest
    mem_limit: 1g
    mem_reservation: 512m

# Java apps: set JVM heap <= container limit (e.g., 75‑80%)
docker run -e JAVA_OPTS="-Xmx768m" --memory=1g my-java-app

# For sustained growth, consider horizontal scaling (multiple instances)

Preventive Measures

Set reasonable memory limits; avoid overly large or tiny values.

For Java/Node.js, explicitly configure heap size.

Configure monitoring and alerts when memory usage exceeds 80% of the limit.

Deploy host‑level OOM alert scripts.

Pitfall 5: Containers Cannot Access the Internet

Symptoms

ping baidu.com

works on host but fails inside container. curl https://google.com times out inside container.

Inter‑container communication works (same bridge network).

Container can reach host IP but not external IPs.

Root Cause

Typical causes include MTU mismatch, iptables NAT rules being cleared, missing host‑to‑container forwarding configuration, or absent proxy settings inside the container.

Diagnostic Commands

# Test connectivity inside container
docker exec <container-id> ping 8.8.8.8
docker exec <container-id> ping baidu.com
docker exec <container-id> curl -v https://google.com

# Check host iptables NAT rules
iptables -t nat -L -n | grep DOCKER

# Inspect Docker bridge configuration
ip addr show docker0
ip route show

# Verify MTU settings
ip link show eth0
docker network inspect bridge | grep -i mtu

# Capture packets for analysis
tcpdump -i docker0 -n host 8.8.8.8

Fixes

MTU Issue

# Set MTU when launching container
docker run --network=host --mtu=9000 my-app

# Or set globally in daemon.json
{ "mtu": 9000 }

iptables Rules Cleared

# Reset Docker iptables rules
iptables -t nat -F
iptables -t filter -F
systemctl restart docker

Proxy Problem

# Check host proxy variables
echo $http_proxy
echo $https_proxy

# Set proxy inside container if needed
docker run -e HTTP_PROXY=http://host.docker.internal:7890 my-app

Pitfall 6: Data Loss After Container Deletion

Symptoms

Data written before redeployment disappears.

Database container restarts with empty database.

Configuration file changes revert after container restart.

Root Cause

By default, a container’s filesystem uses copy‑on‑write; when the container is removed, its layer disappears. Data is not persisted unless stored in a volume, bind mount, or tmpfs.

Diagnostic Commands

# Inspect container mounts
docker inspect <container-id> | grep -A 20 "Mounts"

# List volumes
docker volume ls

# Inspect a specific volume
docker volume inspect <volume-name>

# Verify volume data on host
ls -la /var/lib/docker/volumes/<volume-name>/_data

Fixes

# Named volume for MySQL persistence (docker‑compose)
services:
  mysql:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: "password"
    volumes:
      - mysql_data:/var/lib/mysql
    ports:
      - "3306:3306"

volumes:
  mysql_data:
    driver: local

# Bind mount for configuration files (docker‑compose)
services:
  nginx:
    image: nginx:1.24
    volumes:
      - /data/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - /data/nginx/logs:/var/log/nginx
    ports:
      - "80:80"

Avoid anonymous volumes for critical data; always use named volumes or explicit bind mounts.

Pitfall 7: Latest Tag Confusion

Symptoms

docker run my-app:latest

pulls a new version but behavior is unchanged. docker build -t my-app:1.0 . shows <none> in docker images.

After deployment, the version is unclear.

Root Cause

latest

is just a regular tag; it does not automatically point to the newest image. Its target depends on the last docker build -t …:latest or docker tag operation. Local and remote latest tags may differ.

Diagnostic Commands

# List all tags for an image
docker images my-app

# Show image creation time
docker inspect my-app:latest | grep Created

# Show full image ID
docker images --no-trunc my-app

# Compare local and remote latest
docker pull my-app:latest
docker images my-app:latest

Fixes

# Use explicit version tags instead of latest
FROM nginx:1.24.0-alpine

# Build with precise tag
docker build -t my-app:1.2.3 .
docker build -t my-app:release-20240115 .

docker build -t my-app:v1.2.3-$(git rev-parse --short HEAD) .

Adopt GitOps pipelines that generate unique tags per commit and push them to registries, ensuring traceability.

Pitfall 8: PID 1 Signal Handling Issues

Symptoms

docker stop

times out; container does not stop gracefully.

Container receives SIGTERM but does not exit cleanly. docker kill sends SIGKILL, preventing cleanup.

Logs show main process exited, but child processes become zombies.

Root Cause

PID 1 in a container has special signal handling semantics. If PID 1 is a shell script, the shell does not forward signals to the actual application (e.g., Java). Without proper handling, the container waits for the default 10‑second timeout before being killed.

Diagnostic Commands

# View process tree inside container
docker exec <container-id> ps aux

# Identify PID 1 command line
docker exec <container-id> cat /proc/1/cmdline | tr '\0' ' '

docker exec <container-id> ps -p 1

# Measure stop time
time docker stop <container-id>

Fixes

Option 1: Use exec‑form CMD so the application is PID 1

# Incorrect (shell form)
CMD java -jar app.jar

# Correct (exec form)
CMD ["java", "-jar", "app.jar"]

For scripts, wrap with an entrypoint that forwards signals:

# entrypoint.sh
#!/bin/bash
trap 'kill -TERM $PID' TERM INT
java -jar app.jar &
PID=$!
wait $PID

Option 2: Use Docker’s built‑in init (tini)

# Run with init flag
docker run --init my-app:latest

# Dockerfile example
FROM alpine
RUN apk add --no-cache tini
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["java", "-jar", "app.jar"]

Option 3: Set stop timeout in Swarm/Compose

services:
  app:
    image: my-app:latest
    stop_grace_period: 30s
    stop_signal: SIGTERM

Pitfall 9: Missing Resource Limits Lead to Cascading Failures

Symptoms

Too many containers on a single host exhaust memory.

A Java app with a memory leak drags down all containers.

Containers are OOM‑killed, restart, and OOM again (death loop).

Host load spikes above 100 %, all services become sluggish.

Root Cause

Without explicit limits, a container can consume all host resources. A single misbehaving container can cause OOM across the host, affect the Docker daemon, and trigger kernel‑level OOM, resulting in a system‑wide avalanche.

Diagnostic Commands

# Show memory usage of all containers
docker stats --no-stream

# List containers with their resource limits
docker ps --format "table {{.Names}}	{{.Image}}	{{.Status}}	{{.Ports}}"

# Show memory limits per container
for name in $(docker ps --format "{{.Names}}"); do
  limit=$(docker inspect $name --format '{{.HostConfig.Memory}}')
  echo "$name: $limit"
done

# Host resource overview
top
free -h
df -h

Fixes

# Docker‑compose example with limits
services:
  app:
    image: my-app:latest
    deploy:
      resources:
        limits:
          memory: 512M
          cpus: "0.5"
        reservations:
          memory: 256M
          cpus: "0.25"
      restart_policy:
        condition: on-failure
        max_attempts: 3

# Command‑line run with limits
docker run -d \
  --memory=512m \
  --memory-reservation=256m \
  --cpus=0.5 \
  --cpus-reservation=0.25 \
  --restart=on-failure:3 \
  my-app:latest

Set limits conservatively (e.g., 400 MB needed → 512 MB limit) to leave headroom for the scheduler.

Pitfall 10: Exposed Docker Daemon API (Ports 2375/2376)

Symptoms

Cloud provider alerts that server has port 2375 open. curl http://server:2375/info returns full daemon info. docker -H tcp://server:2375 ps can control remote containers.

Server compromised; attacker uses Docker escape to mine cryptocurrency.

Root Cause

Docker daemon does not listen on TCP by default. Administrators sometimes expose -H tcp://0.0.0.0:2375 for convenience, allowing anyone with network access to control Docker as root. Even the TLS‑enabled 2376 port is unsafe without proper certificates.

Diagnostic Commands

# Check daemon listening ports
ps aux | grep dockerd | grep -v grep
ss -tlnp | grep docker

# Inspect daemon start parameters
systemctl cat docker | grep ExecStart

# Test local exposure
curl http://localhost:2375/info && echo "2375 is open"
curl https://localhost:2376/info && echo "2376 is open"

# External scan (if permitted)
nmap -p 2375,2376 <server-ip>

Fixes

Immediately close exposed Docker API

# If started via systemd, modify unit file
# /etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd

# Do NOT add -H tcp://0.0.0.0:2375

# Reload and restart Docker
systemctl daemon-reload
systemctl restart docker

# Verify ports are closed
ss -tlnp | grep docker

If remote API is required, secure it with TLS

# daemon.json example
{
  "tls": true,
  "tlscert": "/etc/docker/tls/server-cert.pem",
  "tlskey": "/etc/docker/tls/server-key.pem",
  "tlscacert": "/etc/docker/tls/ca.pem",
  "hosts": ["fd://", "tcp://127.0.0.1:2376"]
}

# Client connection
docker -H tcp://server:2376 --tlsverify \
  --tlscert=client-cert.pem \
  --tlskey=client-key.pem \
  --tlscacert=ca.pem ps

Network‑level protection

# Restrict Docker API to management subnet
iptables -A INPUT -p tcp --dport 2375 -s 192.168.1.0/24 -j ACCEPT
iptables -A INPUT -p tcp --dport 2375 -j DROP

Best Practices

Never expose Docker API to the public internet.

Run containers with --read-only when possible.

Use --security-opt=no-new-privileges to prevent privilege escalation.

Avoid --privileged mode in production.

Regularly audit container capabilities: docker inspect --format '{{.HostConfig.CapAdd}}'.

Additional Hidden Pitfalls (5)

Pitfall 11: Container Timezone Issues (repeat of Pitfall 2)

Same causes and fixes as Pitfall 2.

Pitfall 12: Missing --restart Policy

# Recommended restart policy
docker run -d \
  --restart=unless-stopped \
  my-app:latest

# Options:
# no – default, no auto‑restart
# on-failure – restart on non‑zero exit code
# on-failure:3 – max 3 retries
# always – always restart, even after daemon restart
# unless-stopped – restart unless manually stopped

Pitfall 13: Volume Permission Problems

# Example: Nginx container cannot read host directory owned by root
docker run -v /data/www:/usr/share/nginx/html nginx

# Solutions:
# 1. Run container as root (not recommended)
# 2. Adjust host directory permissions
chmod -R 755 /data/www
# 3. Create appropriate user in Dockerfile and set ownership

Pitfall 14: Cross‑Container Network Communication (bridge vs host)

# Preferred: user‑defined bridge network
docker network create my-net
docker run --network=my-net --name redis redis:alpine
docker run --network=my-net --name app my-app:latest
# app can ping redis by name

# Host network shares host namespace; port conflicts may occur
docker run --network=host my-app:latest

Pitfall 15: Multi‑Stage Build Leaking Sensitive Data

# Bad: copying .npm credentials into final image
FROM golang:1.21 AS builder
COPY . /app
RUN go build -o app .
FROM alpine
COPY --from=builder /app /app
COPY --from=builder /root/.npm /root/.npm  # leaks credentials

# Good: only copy final artifact
FROM golang:1.21 AS builder
COPY . /app
RUN go build -ldflags="-w -s" -o app .
FROM alpine
COPY --from=builder /app /app
RUN chmod +x /app
CMD ["/app"]

Conclusion

The ten primary and five secondary Docker pitfalls listed above represent the most common sources of production failures. Prioritizing them by severity and frequency helps teams focus on the most impactful mitigations, such as monitoring disk usage, enforcing resource limits, securing the Docker daemon API, and ensuring proper data persistence.