Top Docker Troubleshooting Tips: Storage, Network, and Runtime Solutions

1 Docker Storage Directory Migration

Default location: Docker stores containers in /var/lib/docker .

Problem: The /var/lib/docker directory grew too large on a server.

Solution 1 – Add a symlink:

# Stop Docker service
sudo systemctl stop docker

# Move directory
sudo mv /var/lib/docker /data/

# Create symlink
sudo ln -s /data/docker /var/lib/docker

# Start Docker service
sudo systemctl start docker

Solution 2 – Edit Docker configuration:

# Method 1 – Edit systemd service file
sudo vim /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --graph=/data/docker/

# Method 2 – Edit daemon.json
sudo vim /etc/docker/daemon.json
{
  "live-restore": true,
  "graph": ["/data/docker/"]
}

Operation notes: Use mv or cp -a to move data, preserving permissions. If moving as root, permission issues are avoided.

2 Docker Device Space Shortage

Increase Docker container size from default 10GB on RHEL7.

Problem: Physical disk is full; Docker reports low available space.

Check disk usage:

df -Th
Filesystem    Size  Used  Avail Use% Mounted on
/dev/vda1      40G   40G    0G 100% /

Check Docker storage info:

docker info
Storage Driver: devicemapper
Data Space Used: 1.225 GB
Data Space Total: 107.4 GB
Data Space Available: 16.78 MB

Solution: Clean logs, remove unused images, or increase Docker base size.

# Show largest container log files
du -d1 -h /var/lib/docker/containers | sort -h
# Truncate log file
cat /dev/null > /var/lib/docker/containers/<container_id>/container_log_name

Increase base size (devicemapper): <code># Edit daemon.json { "live-restore": true, "storage-opt": ["dm.basesize=20G"] } </code> Fix inode exhaustion: Remount with inode64 for ext3 filesystems. <code># Check inode usage df -i # Remount with inode64 sudo mount -o remount,noatime,nodiratime,inode64,nobarrier /dev/vda1 </code>

3 Docker Missing Shared Library

Docker commands need access to /tmp directory.

Problem: docker-compose --version fails with missing libz.so.1.

Solution: Remount /tmp with exec permission.

sudo mount /tmp -o remount,exec

4 Docker Container File Corruption

Improper Docker storage configuration can corrupt container files.

Problem: Container cannot be stopped, restarted, or deleted.

Solution:

# Stop Docker
sudo systemctl stop docker

# Remove container files
sudo rm -rf /var/lib/docker/containers

# Repair metadata
sudo thin_check /var/lib/docker/devicemapper/devicemapper/metadata
sudo thin_check --clear-needs-check-flag /var/lib/docker/devicemapper/devicemapper/metadata

# Start Docker
sudo systemctl start docker

5 Docker Graceful Restart

Keep containers alive when Docker daemon restarts.

Solution: Enable live-restore in daemon configuration.

{
  "live-restore": true
}

# Restart Docker with live-restore
sudo dockerd --live-restore
# Reload daemon
sudo systemctl reload docker

6 Docker Container Cannot Delete

Container files remain after daemon crash.

Problem: docker rm -f fails with conflict error.

Solution: Delete container directory manually.

sudo rm -rf /var/lib/docker/containers/<container_id>
sudo systemctl restart docker.service

7 Docker Container Chinese Locale Issue

MySQL container cannot display Chinese characters.

Problem: Container locale is POSIX, which does not support Chinese.

Solution: Set LANG=C.UTF-8.

# Temporary fix
docker exec -it some-mysql env LANG=C.UTF-8 /bin/bash

# Permanent fix in Dockerfile or run command
docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci

8 Docker Container Network Communication

Understanding Docker's four network models.

Problem: Nginx container proxies to localhost but gets 502 error.

Solution: Use host IP instead of localhost.

# Find host IP (docker0 bridge)
ip addr show docker0
# Use host IP in nginx.conf
proxy_pass http://172.17.0.1:8080;

Note: When using --network=host, localhost refers to the host.

9 Docker Container Bus Error

Bus error occurs due to insufficient shared memory.

Problem: Default shm size is 64M; PyTorch needs more.

Solution: Increase --shm-size.

# Run container with larger shm
docker run -it --rm --shm-size=200m pytorch/pytorch:latest
# In docker-compose
shm_size: '2gb'

Alternative: Ensure enough disk space; low disk also triggers bus error.

10 Docker NFS Mount Error

File locking fails on NFS-mounted volumes.

Problem: fcntl.flock() fails with Bad file descriptor on NFS.

Solution: Use kernel >= 2.6.12 (or 2.6.11+). Upgrade Linux kernel to fix NFSv3/NFSv4 flock issue.

11 Docker Default Network Conflict

Containers use different private address pools causing connectivity issues.

Problem: Some containers get 172.17/172.31, others 192.169/192.168.

Solution: Manually set Docker's default address pool.

{
  "default-address-pools": [{"base":"172.17.0.0/12","size":24}]
}

12 Docker Compose Service Overlap

Docker‑compose projects interfere when using same project name.

Problem: Services from two compose directories restart each other because both have project name app.

Solution: Use distinct directory names or specify project name with -p.

docker-compose -f ./docker-compose.yml -p app1 up -d

13 Docker Exec Command Error in CI

CI jobs fail when using docker exec -it .

Problem: CI environment is not a TTY; -t causes error.

Solution: Remove -t (or -i) when running in non‑interactive CI.

14 Docker Cron Job Error

Backup script fails in crontab due to extra -i flag.

Problem: Using docker exec -it in cron; cron is non‑interactive.

Solution: Use only -t for output or omit both flags for non‑interactive execution.

15 Docker Compose Environment Variable Quoting

Quotes in compose env vars cause values to be unreadable.

Problem: TEST_VAR="test" becomes 'TEST_VAR="test"' after yaml parsing.

Solution: Do not use quotes for simple values; only quote when needed for strings with special characters.

16 Docker Image Deletion Conflict

Cannot delete image because it has dependent child images.

Problem: docker rmi fails even with -f.

Solution: Identify dependent images and remove them first, or delete by tag.

# Inspect dependent images
docker image inspect --format='{{.RepoTags}} {{.Id}} {{.Parent}}' $(docker image ls -q --filter since=<image_id>)
# Remove by tag
docker rmi -f c565xxxxc87f
# Remove dangling images
docker rmi $(docker images --filter "dangling=true" -q --no-trunc)

17 Docker Non‑Root User Permissions

Nginx fails to start due to permission errors on log and cache directories.

Solution: Configure Nginx to use writable paths.

user  www-data;
error_log  /data/logs/master_error.log warn;
pid        /dev/shm/nginx.pid;
client_body_temp_path  /tmp/client_body;
fastcgi_temp_path      /tmp/fastcgi_temp;
proxy_temp_path        /tmp/proxy_temp;
scgi_temp_path         /tmp/scgi_temp;
uwsgi_temp_path        /tmp/uwsgi_temp;

18 Docker IPv6 Binding Issue

Docker tries to bind ports to IPv6 on a host where IPv6 is disabled.

Problem: Error "listen tcp6 [::]:80: socket: address family not supported by protocol".

Solution 1: Bind ports explicitly to IPv4 in compose.

ports:
  - "0.0.0.0:80:80/tcp"

Solution 2: Disable IPv6 in Docker daemon.

{
  "ipv6": false,
  "fixed-cidr-v6": "2001:db8:1::/64"
}
# Reload Docker
systemctl reload docker

Solution 3: Disable IPv6 at kernel level.

echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
systemctl restart networking

19 Docker Compose Hang (Keep Container Alive)

Prevent containers from exiting during debugging.

Solution: Use tty: true and a dummy command.

tty: true
command: tail -f /dev/null
# or entrypoint: tail -f /dev/null

Kubernetes equivalent:

command: ["/bin/bash", "-c", "--"]
args: ["while true; do sleep 30; done;"]

20 Docker Non‑Default Network Subnet

Default Docker subnet conflicts with internal network planning.

Solution: Set a custom address pool.

{
  "default-address-pools": [{"base":"192.168.100.0/20","size":24}]
}
# Restart Docker
systemctl restart docker

Subnet calculation example: Base 10.210.200.0/24 with size:28 gives subnets of 16 IPs each.

21 Add Private Docker Registry

Configure Docker to pull from an insecure private registry.

Solution: Add registry to insecure-registries.

{
  "insecure-registries": ["192.168.31.191:5000"]
}
# Restart Docker
systemctl restart docker
# Login
docker login 192.168.31.191:5000 -u <username> -p <password>