24 Docker Troubleshooting Hacks: Fix Storage, Network, and Startup Issues

1. Relocating Docker storage directory

Docker stores data in /var/lib/docker. To move it to another location (e.g., /data/docker) stop the daemon, move the directory, and either create a symlink or change the daemon configuration.

# Stop Docker
sudo systemctl stop docker
# Move directory
sudo mv /var/lib/docker /data/docker
# Option 1: symlink
sudo ln -s /data/docker /var/lib/docker
# Option 2: daemon config
# /etc/docker/daemon.json
{
  "live-restore": true,
  "graph": "/data/docker"
}
# Reload and start
sudo systemctl daemon-reload
sudo systemctl start docker

2. Resolving “device space shortage”

Check host disk usage ( df -Th) and Docker storage driver info ( docker info). For devicemapper, the default base size may be 10 GB; increase it with --storage-opt dm.basesize=20G or by editing /etc/docker/daemon.json:

{
  "live-restore": true,
  "storage-opt": ["dm.basesize=20G"]
}

Clean large container logs:

# Find biggest log files
du -d1 -h /var/lib/docker/containers | sort -h
# Truncate a log file
cat /dev/null > /var/lib/docker/containers/<strong>CONTAINER_ID</strong>/<strong>LOG_FILE</strong>

If the shortage is caused by exhausted inodes, remount the filesystem with inode64 or enlarge the inode table.

3. Missing shared library (libz.so.1) in docker‑compose

The error occurs because the container cannot execute files from /tmp. Remount /tmp with exec permissions:

sudo mount /tmp -o remount,exec

4. Corrupted container metadata

If a container becomes unmanageable, stop Docker, remove the container’s directory, and repair devicemapper metadata with thin_check:

# Stop Docker
sudo systemctl stop docker
# Remove container files
sudo rm -rf /var/lib/docker/containers/<strong>CONTAINER_ID</strong>
# Repair metadata
sudo thin_check /var/lib/docker/devicemapper/devicemapper/metadata
sudo thin_check --clear-needs-check-flag /var/lib/docker/devicemapper/devicemapper/metadata
# Start Docker
sudo systemctl start docker

5. Graceful daemon restart (live‑restore)

Enable live-restore so containers keep running when the daemon stops:

# /etc/docker/daemon.json
{
  "live-restore": true
}
sudo systemctl reload docker   # or restart if network changes are needed

6. Removing a stuck container

When docker rm fails because the container’s process is missing, delete the container directory manually and restart Docker:

sudo rm -rf /var/lib/docker/containers/<strong>CONTAINER_ID</strong>
sudo systemctl restart docker

7. MySQL container with missing Chinese characters

The default locale inside many containers is POSIX, which does not support UTF‑8. Set LANG=C.UTF-8 or start MySQL with explicit charset options:

# Temporary
docker exec -it <strong>CONTAINER</strong> env LANG=C.UTF-8 bash
# Permanent (MySQL)
docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:latest \
  --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci

8. Container‑to‑host network communication

Inside a container, localhost refers to the container itself. Use the host’s bridge IP (e.g., 172.17.0.1) or run the container in host network mode.

# Find host bridge IP
ip addr show docker0
# Example nginx upstream
proxy_pass http://172.17.0.1:8080;

9. Bus error caused by insufficient shared memory

Docker’s default /dev/shm size is 64 MiB. Increase it with --shm-size or in docker‑compose.yml:

# Run container
docker run -it --rm --shm-size=200m pytorch/pytorch:latest
# docker‑compose
shm_size: '2gb'

10. NFS file‑locking failures

File locking over NFS requires kernel ≥ 2.6.12. On older RHEL/CentOS kernels (e.g., 3.10.0‑693), upgrade the kernel to obtain proper flock() support.

11. Default Docker bridge subnet conflict

Docker’s default bridge pools (172.17.0.0/16 … 172.31.0.0/16) may overlap with existing networks. Define a non‑overlapping pool in /etc/docker/daemon.json:

{
  "default-address-pools": [
    { "base": "192.168.100.0/20", "size": 24 }
  ]
}
sudo systemctl restart docker

12. Docker‑compose project name collision

Compose derives the project name from the directory name. If two compose files reside in directories with the same name, services are considered part of the same project and may restart each other. Resolve by using distinct directory names or the -p flag:

docker-compose -f docker-compose.yml -p app1 up -d

13. CI scripts and docker exec -it

Non‑interactive CI environments cannot allocate a pseudo‑TTY. Remove the -t (or -i) flag when running docker exec in scripts.

14. Cron jobs and Docker exec

In cron, omit -t (and optionally -i) because cron does not provide an interactive terminal.

15. Environment variable quoting in docker‑compose.yml

YAML parsing strips surrounding quotes. Define variables without quotes to avoid unexpected values:

# Correct
environment:
  TEST_VAR: test
# Incorrect (results in literal quotes)
environment:
  TEST_VAR: "test"

16. Deleting images with dependent children

Docker refuses to delete an image that has child images. List dependent images and remove them first, or prune dangling images:

# List dependent images
docker image inspect --format='{{.RepoTags}} {{.Id}} {{.Parent}}' $(docker image ls -q --filter since=<strong>IMAGE_ID</strong>)
# Force‑remove a tag
docker rmi -f <strong>TAG</strong>
# Remove dangling images
docker rmi $(docker images --filter "dangling=true" -q)

17. Adjusting Nginx permissions for non‑root users

When running Nginx as a non‑root user, configure writable paths for logs, PID file, and temporary directories:

user  www-data;
error_log  /data/logs/master_error.log warn;
pid        /dev/shm/nginx.pid;
client_body_temp_path  /tmp/client_body;

18. Binding to IPv6 on hosts with IPv6 disabled

If the host disables IPv6, Docker may try to bind ports on tcp6 and fail. Either bind explicitly to IPv4 (e.g., "0.0.0.0:80:80/tcp") in compose or disable IPv6 in the daemon:

# docker‑compose.yml
ports:
  - "0.0.0.0:80:80/tcp"
# /etc/docker/daemon.json
{
  "ipv6": false,
  "fixed-cidr-v6": "2001:db8:1::/64"
}

19. Docker‑compose HTTP timeout

Long‑running compose operations may exceed the default 60 s timeout. Increase the timeout via environment variables:

export COMPOSE_HTTP_TIMEOUT=500
export DOCKER_CLIENT_TIMEOUT=500

20. Firewall blocking forwarded ports

Firewalld may block ports exposed by Docker. Open the required ports or disable the firewall:

# Open ports
sudo firewall-cmd --permanent --zone=public --add-port=8080/tcp
sudo firewall-cmd --permanent --zone=public --add-port=8081/tcp
sudo firewall-cmd --reload
# Or disable firewalld
sudo systemctl stop firewalld
sudo systemctl disable firewalld

21. Incorrect image tag

“manifest not found” usually means the tag does not exist. Verify the exact tag name (e.g., 0.0.10 vs 0.10).

22. Keeping a container alive

Use tty: true with a dummy command, or set entrypoint: tail to prevent the container from exiting:

services:
  app:
    image: ubuntu:latest
    tty: true
    entrypoint: /usr/bin/tail
    command: "-f /dev/null"

23. Avoiding the default Docker subnet

Configure a custom address pool as shown in section 11 to prevent overlap.

24. Adding an insecure private registry

Configure Docker to trust an insecure registry, restart, and log in:

# /etc/docker/daemon.json
{
  "insecure-registries": ["192.168.31.191:5000"]
}
sudo systemctl restart docker
docker login 192.168.31.191:5000 -u <strong>USERNAME</strong> -p <strong>PASSWORD</strong>