How to Retrieve Crash Logs of a Restarted Kubernetes Pod with kubectl --previous

When a pod is in a crash state and the container keeps restarting, using kubelet logs may fail to capture any output. The solution is to use the kubectl --previous flag, which prints logs from the previous container instance if it exists.

kubectl --previous usage

Single‑container pod: kubectl logs pod-name --previous Multi‑container pod:

kubectl logs pod-name --previous -c container-name

Example output:

NAME                                 READY   STATUS    RESTARTS   AGE
nginx-7d8b49557c-c2lx9               2/2     Running   5          

kubectl logs nginx-7d8b49557c-c2lx9 --previous
Error: xxxxxxxxxxx

kubelet keeps the logs of the most recent failed containers under /var/log/pods/podname. These are symbolic links to the Docker container log files. A separate symlink points to the log file of the previous crashed container, which is what the --previous option reads.

To view a pod and its log files on the node:

ubuntu@~$ kubelet get pod
NAME        READY   STATUS    RESTARTS   AGE
busybox     1/1     Running   2394       99d
nginx-deployment-6wlhd 1/1 Running 0 79d
redis       1/1     Running   0          49d

List the log files for a pod on its node:

ls /var/log/pods/default_busybox_f72ab71a-5b3b-4ecf-940d-28a5c3b30683/busybox
2393.log  2394.log

The numbers indicate the restart count; 2393.log is the log after the 2393rd restart, and 2394.log after the 2394th.

These files are symlinks to Docker logs, for example:

/busybox# stat 2393.log
File: 2393.log -> /data/kubernetes/docker/containers/68a5b32c9fdb.../68a5b32c9fdb...-json.log

/busybox# stat 2394.log
File: 2394.log -> /data/kubernetes/docker/containers/2ed9ebf05852.../2ed9ebf05852...-json.log

Reading logs without --previous accesses the current container’s log file, while adding --previous reads the previous container’s log file because kubelet retains a symlink to it.

Manual editing of these log files confirms the behavior:

/busybox# cat 2393.log
{"log":"last crash logs
","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}

/busybox# cat 2394.log
{"log":"now pod log
","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}

kubectl logs busybox --previous
last crash logs

kubectl logs busybox
now pod log

Creating regular files in place of the symlinks and testing again shows that --previous reads the file pointed to by the previous‑container symlink.

Conclusion: kubelet reads log files from /var/log/pods/. The --previous flag also reads from this directory, using a dedicated symlink that points to the log of the last exited container, allowing you to retrieve logs from before a crash.

Link: https://blog.csdn.net/qq_43684922/article/details/128881716