How I Uncovered Two Hidden Causes of Remote Docker Container Restarts

Background

Rust backend built with a three‑stage cargo‑chef Dockerfile, orchestrated by Docker Compose, stored in an Alibaba Cloud ACR registry, and deployed to a remote host via a Python script that runs docker build, pushes the image, then SSHes to the host and executes docker‑compose pull && docker‑compose up -d. After deployment the container repeatedly entered the Restarting state.

Investigation pivot: use docker inspect ExitCode

Because docker logs were empty (the container crashed before the logging system started), the author inspected the container state with:

docker inspect <container> --format "State={{.State.Status}} ExitCode={{.State.ExitCode}} RestartCount={{.State.RestartCount}}"

The observed exit‑code sequence 0 → 101 indicated two distinct failure modes. Common exit‑code meanings:

0 : program exited normally – often an empty binary or premature termination.

101 : Rust panic – typically configuration validation failure (missing env var, unreadable file).

127 : command not found – wrong binary path or missing dynamic library.

137 : SIGKILL – OOM kill.

139 : segmentation fault – usually glibc version mismatch.

Root cause 1: corrupted build cache produced an empty binary

Symptoms and diagnosis

docker logs

were empty, but docker inspect showed:

State=restarting ExitCode=0 RestartCount=13
StartedAt and FinishedAt differ by only 0.14 s

ExitCode 0 with an immediate exit meant the main function returned without executing any business logic. To verify the artifact, the binary was extracted from the image and examined:

# Size check – a normal Rust release binary is >10 MB
ls -la /usr/local/bin/<binary>
# This build was only 438 KB → severely undersized

# Symbol check – look for async runtime symbols
strings <binary> | grep -c "tokio"   # 0 (should be thousands)
strings <binary> | grep -c "0.0.0.0" # 0 (no listen address)
strings <binary> | grep -c "<your‑route>" # 0 (no routes)

The binary lacked tokio symbols and any network‑related strings, confirming it was an empty placeholder.

Why the cache became polluted

The Dockerfile uses three stages:

# Stage planner – generate recipe.json
FROM chef AS planner
COPY Cargo.toml Cargo.lock ./
COPY src ./src
RUN cargo chef prepare --recipe-path recipe.json

# Stage builder – cache dependencies, then compile business code
FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json   # cacheable deps
COPY src ./src                                            # business source
RUN cargo build --release --locked                       # cacheable app

Docker caches a layer as long as its inputs (Dockerfile instruction + file fingerprints) are unchanged. If a previous build produced a bad artifact—e.g., an incomplete source tree or an out‑of‑sync recipe.json —the RUN cargo build layer cached the faulty binary. Subsequent builds reused that polluted layer, repeatedly emitting the same 438 KB empty binary while the build logs still reported “Finished release profile” and exited successfully.

Fix

Rebuilding without any cache forces a fresh compilation:

docker build --no-cache -t app:local .
# Resulting binary size ~18 MB, tokio symbols appear 3227 times → correct artifact

After confirming the correct binary, the image was retagged, pushed to ACR, and redeployed.

Preventive automation

The deployment script now extracts the binary from a temporary container, checks its size, and aborts the push if the size is below a threshold (e.g., 2 MB), indicating a possible empty shell:

# Create container quietly and capture its ID
docker create --quiet <image> > /tmp/c.cid
cid=$(cat /tmp/c.cid)
# Copy binary out of the container
docker cp $cid:/usr/local/bin/app /tmp/c.bin
# Measure size
size=$(stat -c%s /tmp/c.bin)
if [ $size -lt $((2*1024*1024)) ]; then
  echo "Suspicious empty artifact (${size}B), aborting push"
  exit 1
fi
# Clean up
docker rm $cid >/dev/null
rm -f /tmp/c.bin

Lesson 1: Docker layer cache can silently corrupt builds; a polluted layer will repeatedly produce incorrect artifacts without any error messages. Verify the produced artifact (size, key symbols) before publishing.

Root cause 2: file permission and UID mismatch in bind‑mounted secrets

Symptoms and diagnosis

After fixing the binary, the container still restarted, now with ExitCode 101. Logs showed:

thread 'main' panicked at src/config.rs:
read private key failed (/app/.../key.pem): Permission denied (os error 13)

The error changed from “file not found” to “Permission denied”, indicating the file existed but was unreadable inside the container.

Root cause

The deployment script uploaded the private key via SFTP as root (mode 600). Docker Compose then bind‑mounted the file read‑only into the container, which runs as the non‑root user appuser (uid=1001) (defined in the Dockerfile). Because the host‑side file remained owned by root, appuser could not read it:

# Host file owner
ls -la .../key.pem
# -rw------- 1 root root … key.pem

# Container user
docker run --rm <image> id
# uid=1001(appuser)

Fix

Align the file’s UID/GID with the container user before mounting:

# Ensure the Dockerfile creates appuser with uid 1001
# Change ownership on the host, keep restrictive mode
chown -R 1001:1001 /path/to/keys && chmod 600 /path/to/keys/*.pem

After applying chown, the container could read the key and start normally.

Lesson 2: When a container runs as a non‑root user, any bind‑mounted file must have its host‑side owner UID match the container UID; otherwise the container sees “Permission denied” even though host‑side permissions appear correct.

Combined observations

Both failures manifested only at runtime on the remote host while the local build and push pipelines reported success:

Cache pollution produced an empty binary – the only clue was ExitCode 0 with immediate exit and no logs.

UID/owner mismatch on a bind‑mounted secret caused a Rust panic (ExitCode 101) due to permission denial.

These “green‑pipeline, red‑runtime” failures are hard to detect because automated checks pass. Mitigation consists of embedding artifact validation (size, symbol checks) in the deployment script and ensuring UID alignment for any files mounted into non‑root containers.