Mastering Kubernetes Liveness & Readiness Probes: Health Checks, Debugging, and Remote Troubleshooting

Why Application Health Matters in Kubernetes

After moving an application to Kubernetes, maintaining its health and stability is essential. Two key aspects are improving observability (monitoring health status, resource usage, and logs) and enhancing recoverability (automatic self‑healing and restart mechanisms).

Liveness and Readiness Probes Overview

Liveness probes determine whether a container is alive; if the probe fails, kubelet kills the pod and the restart policy decides whether to restart it.

Readiness probes assess whether a pod is ready to serve traffic; if the probe fails, the pod is removed from the service endpoints, preventing traffic from being routed to it.

Probe Types

httpGet : Sends an HTTP GET request; any status code between 200‑399 is considered healthy.

Exec : Executes a command inside the container; an exit code of 0 indicates health.

tcpSocket : Attempts a TCP connection to the specified IP and port; a successful connection means the container is healthy.

Probe Results

Success : The container passed the health check.

Failure : The container failed the check; for Readiness the pod is removed from endpoints, for Liveness the pod is killed and possibly restarted.

Unknown : The probe timed out or did not return a result; no action is taken until the next check.

Pod Probe Specification (PodSpec)

A typical PodSpec includes the three probe types and several global parameters: initialDelaySeconds: Delay before the first check (useful for slow‑starting apps). periodSeconds: Interval between checks (default 10 s). timeoutSeconds: Timeout for a single check. successThreshold: Number of consecutive successes required to consider the pod healthy. failureThreshold: Number of consecutive failures before marking the pod unhealthy (default 3).

Example YAML snippets (shown in the original slides) illustrate how to configure an exec probe that runs cat /tmp/healthy, an httpGet probe with path and port, and a tcpSocket probe on port 8080.

Common Pod Issues and Diagnosis

Typical problematic pod states include:

Pending : Scheduler cannot place the pod (resource constraints, node selector, etc.). Use kubectl describe pod to view events.

Waiting : Image pull failures (private image without secret, missing image, network issues).

CrashLoopBackOff : Pod repeatedly restarts; investigate container logs.

Running but not serving : Misconfiguration (e.g., wrong labels, ports, or probe settings) prevents the service from exposing the pod.

Diagnosis steps:

Run kubectl describe pod <name> to see status and events.

Check kubectl get pod <name> -o yaml for probe configuration and resource definitions.

Inspect logs with kubectl logs <pod> -c <container>.

Remote Debugging Strategies

Two main scenarios are covered:

Debugging a pod : Use

kubectl exec -it <pod> -c <container> -- /bin/bash

to open an interactive shell inside a specific container.

Debugging a service :

For reverse traffic (local app to remote service), Telepresence can proxy a local process to a remote service. Deploy the Telepresence proxy in the remote cluster and run telepresence --swap-deployment <deployment> to replace the remote deployment with the local one.

For forward traffic (local debugging of a remote service), use

kubectl port-forward svc/<service> <localPort>:<remotePort>

to expose the remote service locally.

kubectl‑debug

kubectl‑debug is a plugin that creates an additional debug container sharing the same Linux namespace as the target pod. The debug container can include tools such as htop, netstat, or custom binaries. After launching, you can run diagnostic commands inside the debug container without affecting the original application. When finished, exit the debug session to terminate the debug pod.

Key Takeaways

Liveness probes ensure a pod is alive; Readiness probes ensure a pod is ready to receive traffic.

Proper probe configuration (type, thresholds, delays) is critical to avoid false positives/negatives.

Diagnosing pod issues follows a three‑step process: describe, inspect configuration, and check logs/events.

Remote debugging can be achieved with kubectl exec, kubectl port-forward, Telepresence, or kubectl‑debug, each suited to different scenarios.