How to Properly View Container Logs Without Using Tail -f Inside Pods

Background and Problem

In containerized environments log collection and inspection are frequent operations tasks. Using tail -f inside a container works only for very small clusters; as the number of containers grows the approach becomes inefficient and error‑prone.

1. Fundamentals of Container Logging

1.1 Log storage location

With containerd the logs are stored on the node under

/var/log/pods/<namespace>/<pod_name>/<container_name>/

. Each file follows the pattern <container_name>/<restart_count>.log where restart_count starts at 0 and increments on each restart.

# List log directories on the node
ls -la /var/log/pods/

# List a specific pod's logs
ls -la /var/log/pods/default_nginx-deployment-6d9f4c8b7-x2m9n_abc123/

# List a specific container's log file
ls -la /var/log/pods/default_nginx-deployment-6d9f4c8b7-x2m9n_abc123/nginx/

# Show the log content
cat /var/log/pods/default_nginx-deployment-6d9f4c8b7-x2m9n_abc123/nginx/0.log

1.2 kubelet log handling

kubelet redirects the container’s stdout and stderr to log files. It also rotates logs based on size using the defaults containerLogMaxSize: 10Mi and containerLogMaxFiles: 5.

# Show the log rotation configuration
cat /var/lib/kubelet/config.yaml | grep -A 5 "logging"

# Default values (shown as comments)
# containerLogMaxSize: 10Mi
# containerLogMaxFiles: 5

1.3 Role of containerd‑shim

containerd‑shim

is an intermediate process that acts as the container’s parent, handles exit signals, keeps the container alive after kubelet restarts, and manages stdin/stdout streams.

# List all containerd‑shim processes
ps aux | grep containerd‑shim

# List containers managed by containerd‑shim
sudo ctr -n k8s.io containers list

2. Correct Use of kubectl logs

2.1 Basic usage

# Show the latest logs of a pod
kubectl logs nginx-deployment-6d9f4c8b7-x2m9n

# Show logs of the previous terminated container (after a restart)
kubectl logs nginx-deployment-6d9f4c8b7-x2m9n --previous

# Follow logs (similar to tail -f)
kubectl logs -f nginx-deployment-6d9f4c8b7-x2m9n

# Show the last 100 lines
kubectl logs nginx-deployment-6d9f4c8b7-x2m9n --tail=100

# Show logs from the last hour
kubectl logs nginx-deployment-6d9f4c8b7-x2m9n --since=1h

# Show logs since a specific timestamp
kubectl logs nginx-deployment-6d9f4c8b7-x2m9n --since-time="2026-01-01T00:00:00Z"

2.2 Logs from multi‑container Pods

# View logs of a specific container in a multi‑container pod
kubectl logs <pod-name> -c <container-name>

# Follow logs of that container
kubectl logs -f <pod-name> -c <container-name>

# View logs of all containers in the pod
kubectl logs <pod-name> --all-containers=true

# Use a label selector to view logs from multiple pods
kubectl logs -l app=nginx --tail=100

2.3 Common errors and solutions

Pod not found

# Command that fails
kubectl logs nginx
Error from server (NotFound): pods "nginx" not found

# Fix: specify the correct namespace
kubectl logs nginx -n production
kubectl logs nginx -n production --all-namespaces

Container not found

# Command that fails
kubectl logs multi-container-pod
Error from server (BadRequest): a container name must be specified for pod multi-container-pod, choose one of: [nginx sidecar]

# Fix: use -c to specify the container
kubectl logs multi-container-pod -c nginx
kubectl logs multi-container-pod -c sidecar

Pod in Terminating state

# Command that fails
kubectl logs nginx-deployment-6d9f4c8b7-x2m9n
Error from server (NotFound): pods "nginx-deployment-6d9f4c8b7-x2m9n" not found

# Fix: use --previous to view the previous container's logs
kubectl logs nginx-deployment-6d9f4c8b7-x2m9n --previous

# If the pod has been deleted, view the log file directly on the node
ssh <node-name> "cat /var/log/pods/<pod-id>/<container-name>/0.log"

3. Log Filtering and Formatting

3.1 Level filtering

# Show only ERROR level logs
kubectl logs nginx-deployment-6d9f4c8b7-x2m9n | grep ERROR

# Show ERROR/FATAL/CRITICAL logs from the last hour
kubectl logs nginx-deployment-6d9f4c8b7-x2m9n --since=1h | grep -E "ERROR|FATAL|CRITICAL"

# Count logs per level
kubectl logs nginx-deployment-6d9f4c8b7-x2m9n --since=24h | awk '{print $5}' | sort | uniq -c

3.2 Timestamp filtering

# Extract logs within a specific time range (awk example)
kubectl logs nginx-deployment-6d9f4c8b7-x2m9n --since=1h | \
  awk -F'[] []' '$2 " " $3 >= "2026-01-15 10:00:00" && $2 " " $3 <= "2026-01-15 11:00:00"'

# Use sed for time‑range filtering
kubectl logs nginx-deployment-6d9f4c8b7-x2m9n | \
  sed -n '/2026-01-15 10:00:00/,/2026-01-15 11:00:00/p'

3.3 JSON log parsing with jq

# Install jq inside the pod (Alpine example)
kubectl exec -it <pod-name> -- apk add jq

# Extract level and message fields
kubectl logs <pod-name> | jq -r '.level, .message'

# Count occurrences of each error type
kubectl logs <pod-name> --since=24h | jq -r '.error // "no-error"' | sort | uniq -c | sort -rn

# Extract timestamp and message for error level logs
kubectl logs <pod-name> | jq -r 'select(.level == "error") | .timestamp, .message'

4. Node‑Level Log Access

4.1 Directly access log files

# SSH to the node
ssh <node-name>

# List pod log directories
ls -la /var/log/pods/

# View container logs with journalctl
sudo journalctl -u kubelet --since "1 hour ago" | grep <pod-name>

# Use ctr to view container information and logs
sudo ctr -n k8s.io containers list | grep <container-name>
sudo ctr -n k8s.io tasks logs <container-id>

4.2 containerd logs

# View containerd service logs
sudo journalctl -u containerd --since "1 hour ago" | grep -E "error|warn" --color=never

# Check containerd service status
sudo systemctl status containerd

# Inspect containerd config for sandbox or log_level settings
cat /etc/containerd/config.toml | grep -E "sandbox|log_level"

4.3 kubelet logs

# View recent kubelet logs
sudo journalctl -u kubelet --since "30 minutes ago" --no-pager

# Filter error lines
sudo journalctl -u kubelet --since "1 hour ago" | grep -i error

# Follow kubelet logs in real time
sudo journalctl -u kubelet -f

5. Distributed Log‑Collection Architecture

5.1 EFK stack

EFK (Elasticsearch + Fluent Bit + Kibana) is a common solution for Kubernetes log aggregation.

Fluent Bit : runs as a DaemonSet on each node, collects container logs and forwards them to Elasticsearch.

Elasticsearch : stores and indexes log data.

Kibana : UI for querying and visualising logs.

5.1.1 Deploy Fluent Bit as a DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit
  template:
    metadata:
      labels:
        k8s-app: fluent-bit
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:3.0.0
        ports:
        - name: http
          containerPort: 2020
        volumeMounts:
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: kube-system
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        5
        Log_Level    info
        Daemon       off
        Parsers_File parsers.conf
    [INPUT]
    Name   tail
    Path   /var/log/containers/*.log
    Parser docker
    Tag    kube.containers.*
    Refresh_Interval 5
    [OUTPUT]
    Name   es
    Match  kube.containers.*
    Host   elasticsearch.logging.svc
    Port   9200
    Index  kubernetescontainers
    Type   flb_type
    Retry_Limit 2
  parsers.conf: |
    [PARSER]
        Name   docker
        Format json
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L

5.1.2 Fluent Bit configuration details

# Add Kubernetes metadata filter
[FILTER]
Name                kubernetes
Match               kube.containers.*
Kube_URL            https://kubernetes.default.svc:443
Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix     kube.containers.var.log.containers.
Merge_Log           On
Keep_Log            Off
K8S-Logging.Parser  On
K8S-Logging.Exclude On

5.2 Loki stack

Loki, developed by Grafana Labs, is a low‑resource log aggregation system with a Prometheus‑like architecture.

# Loki deployment example
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app: loki
  template:
    metadata:
      labels:
        app: loki
    spec:
      containers:
      - name: loki
        image: grafana/loki:3.0.0
        ports:
        - name: http
          containerPort: 3100
        args:
        - -config.file=/etc/loki/local-config.yaml
        - -target=all
        resources:
          limits:
            cpu: "2"
            memory: 4Gi
          requests:
            cpu: "500m"
            memory: 1Gi
---
# Promtail (log collector) deployment
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: promtail
  template:
    metadata:
      labels:
        app: promtail
    spec:
      containers:
      - name: promtail
        image: grafana/promtail:3.0.0
        args:
        - -config.file=/etc/promtail/promtail.yaml
        volumeMounts:
        - name: config
          mountPath: /etc/promtail
        - name: varlog
          mountPath: /var/log
        - name: pods
          mountPath: /var/log/pods
      volumes:
      - name: config
        configMap:
          name: promtail-config
      - name: varlog
        hostPath:
          path: /var/log
      - name: pods
        hostPath:
          path: /var/log/pods

5.3 Log‑collection best practices

5.3.1 Structured logs

{
  "timestamp": "2026-01-15T10:30:00.123Z",
  "level": "error",
  "message": "Database connection failed",
  "service": "user-api",
  "request_id": "abc123",
  "error_code": "DB_CONN_001",
  "stack_trace": "..."
}

5.3.2 Log level conventions

ERROR : functional impact, requires immediate attention.

WARN : potential issue or misconfiguration.

INFO : important business events or state changes.

DEBUG : development‑debug information; should be disabled in production.

5.3.3 Retention policy

Define retention based on compliance and storage cost:

Hot storage (Elasticsearch/Loki): 7‑30 days

Warm storage (object storage): 30‑90 days

Cold storage (archival): >90 days

6. Application‑level Logging Best Practices

6.1 Log output to stdout/stderr

# Python example
import logging, sys
logging.basicConfig(
    level=logging.INFO,
    format='{"timestamp": "%({asctime}s)", "level": "%({levelname}s)", "message": "%({message}s)"}',
    handlers=[logging.StreamHandler(sys.stdout)]
)

6.2 Log rotation configuration

# Logback (Java) example
<configuration>
  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>/var/log/app/application.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>/var/log/app/application.%d{yyyy-MM-dd}.log</fileNamePattern>
      <maxHistory>30</maxHistory>
      <totalSizeCap>10GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <pattern>{"timestamp":"%d{ISO8601}","level":"%level","logger":"%logger","message":"%msg"}%n</pattern>
    </encoder>
  </appender>
  <root level="INFO">
    <appender-ref ref="FILE"/>
  </root>
</configuration>

6.3 Sidecar log‑collection pattern

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-with-sidecar
spec:
  template:
    spec:
      containers:
      - name: app
        image: my-app:latest
        volumeMounts:
        - name: log-volume
          mountPath: /var/log/myapp
      - name: log-collector
        image: fluent/fluent-bit:3.0.0
        volumeMounts:
        - name: log-volume
          mountPath: /var/log/myapp
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc
      volumes:
      - name: log-volume
        emptyDir: {}
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config

7. Log‑related Troubleshooting

7.1 kubectl logs hangs

# Check API server health
kubectl get componentstatuses

# Inspect API server logs
kubectl logs -n kube-system kube-apiserver-<node-name> --tail=100

# Verify kubelet status on the node
ssh <node-name> "systemctl status kubelet"

# Bypass API server and read the log file directly
ssh <node-name> "cat /var/log/pods/<namespace>_<pod-name>_<uid>/<container-name>/0.log | tail -100"

7.2 Log loss

7.2.1 Container restart deletes old logs

By default logs from a restarted container are removed. Increase retention by adjusting kubelet configuration.

# Increase kubelet log retention
cat /var/lib/kubelet/config.yaml

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerLogMaxFiles: 10
containerLogMaxSize: 50Mi

7.2.2 Fluent Bit/Fluent d not collecting logs

# Check Fluent Bit pod status
kubectl get pod -n kube-system -l k8s-app=fluent-bit

# View Fluent Bit logs
kubectl logs -n kube-system -l k8s-app=fluent-bit

# Verify ConfigMap configuration
kubectl describe configmap fluent-bit-config -n kube-system

# Test Fluent Bit configuration inside the pod
kubectl exec -it -n kube-system fluent-bit-xxxx -- fluent-bit --test --parser=parsers.conf < /var/log/containers/*.log

7.3 Oversized log files

# Find log files larger than 100 MiB
find /var/log/pods -name "*.log" -size +100M -exec ls -lh {} \;

# Compress a large log file manually
sudo gzip /var/log/pods/<path>/0.log

# Delete old rotated logs (>7 days)
sudo find /var/log/pods -name "*.log.*" -mtime +7 -delete

# Monitor log directory size
du -sh /var/log/pods/*

7.4 Inconsistent logs between kubectl logs and node files

# Compare kubectl output with the file on the node
kubectl logs <pod-name> > /tmp/kubectl.log
ssh <node-name> "cat /var/log/pods/<pod-id>/<container>/0.log" > /tmp/file.log
diff /tmp/kubectl.log /tmp/file.log

# Possible reasons:
# 1. Log rotation changed the underlying file.
# 2. Container restart – --previous points to a different file.
# 3. Encoding issues.

8. Log Monitoring and Alerting

8.1 Error‑log monitoring with Prometheus

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: log-error-monitoring
  namespace: monitoring
spec:
  groups:
  - name: application-logs
    rules:
    - alert: HighErrorRate
      expr: |
        sum(rate(container_log_messages_total{level="error"}[5m])) by (namespace, pod) > 0.1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High error rate in {{ $labels.namespace }}/{{ $labels.pod }}"
        description: "Error rate exceeds 10% over 5 minutes"

8.2 Log storage capacity alerts

- alert: LogStorageUsageHigh
  expr: |
    (node_filesystem_avail_bytes{mountpoint="/var/log"} / node_filesystem_size_bytes{mountpoint="/var/log"}) < 0.2
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Log storage usage is above 80%"

9. Practical Command Reference

9.1 View logs from multiple Pods quickly

# Show logs of all Pods with label app=nginx (last 50 lines)
kubectl logs -l app=nginx --tail=50

# Show error logs from the last hour
kubectl logs -l app=nginx --since=1h | grep -E "ERROR|FATAL"

# Count error occurrences in the last 24 hours
kubectl logs -l app=nginx --since=24h | grep -c "ERROR"

9.2 Real‑time multi‑Pod log view with stern

# Follow logs of all nginx Pods in namespace production for the last 10 minutes
stern -n production app=nginx --since=10m

# Show only lines containing ERROR
stern -n production app=nginx --filter=ERROR

# Include timestamps
stern -n production app=nginx --timestamp

9.3 Common log‑analysis commands

# Distribution of log levels
kubectl logs <pod-name> | awk '{print $5}' | sort | uniq -c | sort -rn

# Top error types (JSON logs)
kubectl logs <pod-name> --since=24h | jq -r '.error // "no-error"' | sort | uniq -c | sort -rn | head -10

# Average request latency (JSON logs)
kubectl logs <pod-name> | jq -r 'select(.type=="request") | .duration' | awk '{sum+=$1; count++} END {print "Average:", sum/count, "ms"}'

# Extract logs within a specific time window
kubectl logs <pod-name> | awk -F'[] []' '$2 >= "2026-01-15T10:00:00" && $2 <= "2026-01-15T11:00:00"'

Conclusion

Proper handling of container logs consists of three layers:

Fundamental layer : master kubectl logs usage, including --tail, --since, -f, --previous, and specifying container names for multi‑container Pods.

Architecture layer : understand where logs are stored ( /var/log/pods), how kubelet rotates logs, and the role of containerd‑shim.

Platform layer : build a distributed log‑collection stack (EFK, Loki) for centralized storage, search, and analysis.

Avoid exec‑ing into containers to read logs; rely on kubectl logs or a centralized logging system. Standardise log output (JSON), use clear log levels, and keep consistent field naming to enable downstream analysis and rapid issue localisation.