Debugging Kubernetes Container Networks with nsenter and Temporary Pods

In a Kubernetes cloud environment, containers often lack network debugging tools such as tcpdump, making packet capture difficult. The nsenter utility can enter a specific namespace and is commonly used for container debugging.

Debugging Container Network

Using nsenter, you can easily enter the container's network namespace from the host with the following command:

# 设置containerid
containerid=xxx
# 获取容器主进程
pid=$(docker inspect -f {{.State.Pid}} $containerid)
# 进入容器networker namespace
nsenter -n --target $pid

After entering, you can run host tools such as tcpdump, netstat, and other networking commands inside the container.

Logging into a Kubernetes Node

If you only have Apiserver permissions, you can still log into a node using nsenter. A temporary login script is shown below (requires privileged and hostPID permissions):

node=xxx
cmd='[ "nsenter", "--target", "1", "--mount", "--uts", "--ipc", "--net", "--pid", "--"]'
overrides="$(cat <<EOT
{
  "spec": {
    "nodeName": "$node",
    "hostPID": true,
    "hostNetwork": true,
    "containers": [
      {
        "securityContext": {"privileged": true},
        "image": "alpine",
        "name": "nsenter",
        "stdin": true,
        "stdinOnce": true,
        "tty": true,
        "command": $cmd
      }
    ],
    "tolerations": [{"operator": "Exists"}]
  }
}
EOT)"

pod="kube-nodeshell-$(env LC_ALL=C tr -dc a-z0-9 </dev/urandom | head -c 6)"
kubectl run --image=alpine --restart=Never --rm --overrides="$overrides" -it $pod

The principle is that hostPID=true shares the host's PID namespace, allowing the container to see all host processes; then nsenter enters the host's mount, uts, ipc, net, and pid namespaces to obtain a shell similar to the host.

If you need frequent access, you can deploy a DaemonSet that runs an nsenter pod on each node (recommended only for testing environments due to security risks):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-nodehsell
  labels:
    app: kube-nodehsell
spec:
  selector:
    matchLabels:
      app: kube-nodehsell
  template:
    metadata:
      labels:
        app: kube-nodehsell
    spec:
      tolerations:
      - operator: "Exists"
      containers:
      - name: kube-nodehsell
        image: alpine
        command:
        - nsenter
        - --target
        - "1"
        - --mount
        - --uts
        - --ipc
        - --net
        - --pid
        - --
        - sleep
        - infinity
        securityContext:
          privileged: true
      hostIPC: true
      hostPID: true
      hostNetwork: true
      priorityClassName: system-node-critical

All related files are available in the kube-nodeshell repository.

Temporary Containers

Since Kubernetes 1.18, temporary containers can be added to a pod using the kubectl debug command, which also allows logging into a node shell for simple debugging tasks. Compared with the nsenter method, kubectl debug only shares the pid and hostNetwork, while nsenter provides more flexibility to use host tools and perform privileged operations.

-END-