Best Practices for Container Operations: Logging, Monitoring, Security, and Immutability

The article presents a collection of best practices that make containers easier to operate, covering security, monitoring, logging, and design principles inspired by the 12‑factor app methodology.

Use Container Native Logging – Containers should write logs to stdout and stderr , allowing Docker to capture them and exposing them via docker logs . In Kubernetes, logs can be centralized with Fluentd and Stackdriver Logging, or with an EFK stack.

JSON Logging – Log management systems store time‑indexed documents, often in JSON. Converting a log line such as [2018-01-01 01:01:01] foo - WARNING - foo.bar - There is something wrong. to JSON yields { "date": "2018-01-01 01:01:01", "component": "foo", "subcomponent": "foo.bar", "level": "WARNING", "message": "There is something wrong." } which enables field‑based search. Each JSON line must be a complete event, e.g. {"date":"2018-01-01 01:01:01","component":"foo","subcomponent":"foo.bar","level":"WARNING","message":"There is something wrong."} .

Sidecar Logging Aggregator – For applications that write logs to files (e.g., Tomcat), run a sidecar container with a logging agent that shares an emptyDir volume. The sidecar reads the files and forwards logs to the central system. Example YAML can be found in the Kubernetes contrib repository.

Ensure Containers Are Stateless and Immutable – Store all persistent data outside the container using Cloud Storage, Redis/Memcached, or Persistent Disks. Treat containers as immutable; any change requires building a new image and redeploying. Externalize configuration via environment variables, Secrets, or ConfigMaps, and update by deploying a new container.

Avoid Privileged Containers – Privileged containers bypass most security controls. Prefer using securityContext or Docker --cap-add to grant only needed capabilities, or move host‑level changes to a sidecar or init container. PodSecurityPolicy can forbid privileged pods.

Make Applications Easy to Monitor – Use Prometheus or Stackdriver for white‑box monitoring. Expose metrics on a /metrics HTTP endpoint, e.g.: http_requests_total{method="post",code="200"} 1027 http_requests_total{method="post",code="400"} 3 http_requests_total{method="get",code="200"} 10892 http_requests_total{method="get",code="400"} 97 Sidecar containers can also export metrics for applications that do not natively provide a /metrics endpoint.

Expose Application Health – Implement liveness ( /health ) and readiness ( /ready ) probes. Liveness confirms the container is running and its dependencies are met; readiness indicates the app can receive traffic. Kubernetes uses readiness probes to orchestrate rolling updates.

Avoid Running as Root – Running as root increases the risk of container escape. Use runAsUser in the pod spec or a non‑root USER in the Dockerfile. Test locally with a random user: docker run --user $((RANDOM+1)) [YOUR_CONTAINER] Adjust fsGroup for external volumes if needed.

Carefully Choose Image Tags – Avoid the mutable latest tag. Prefer immutable tags (e.g., debian:9.4 ) or semantic version tags ( myapp:1.2.3 ) to ensure reproducible builds and deployments. When a new patch is released, update the tag (e.g., myapp:1.2 ) to pull the latest patch automatically.