Mastering nsenter: Debug Container Networks and Namespaces with Ease

nsenter is a command from the util-linux package that runs a specified program inside the namespace of a given process.

Use Cases

A typical use case is entering a container's network namespace to run host tools such as

ip address

,

ping

,

telnet

, or

tcpdump

, avoiding the need to inspect the container IP via

docker inspect

. nsenter can also enter other namespaces such as

mnt

,

uts

,

ipc

,

pid

, and

user

, and can set a root directory and working directory.

Usage

Syntax:

<code>nsenter [options] [program [arguments]]

options:
  -t, --target pid          Target process PID whose namespace to enter
  -m, --mount[=file]        Enter mount namespace (optional file)
  -u, --uts[=file]          Enter UTS namespace
  -i, --ipc[=file]          Enter IPC namespace
  -n, --net[=file]          Enter network namespace
  -p, --pid[=file]          Enter PID namespace
  -U, --user[=file]         Enter user namespace
  -G, --setgid gid          Set GID for the program
  -S, --setuid uid          Set UID for the program
  -r, --root[=directory]    Set root directory
  -w, --wd[=directory]      Set working directory

If no program is given, the default is $SHELL.</code>

Example:

<code># Get the PID of an nginx container
[root@host ~]# docker inspect -f {{.State.Pid}} nginx
5645

# Enter the container's network namespace
[root@host ~]# nsenter -n -t5645
[root@host ~]# ip addr
... (output showing interfaces) ...
</code>

In Kubernetes, obtain the container ID first:

<code># Get containerID via yaml output
[root@node1 test]# kubectl get pod test -oyaml | grep containerID
  - containerID: docker://cf0873782d58...

# Or more precisely
[root@node1 test]# kubectl get pod test -o template --template='{{range .status.containerStatuses}}{{.containerID}}{{end}}'
docker://cf0873782d58...
</code>

Principle

Namespace

Linux namespaces isolate various resources of processes. Types include:

mount : separate mount points (since Linux 2.4.19)

ipc : separate IPC objects like message queues, shared memory, semaphores (since Linux 2.6.19)

uts : separate hostname and domain name (since Linux 2.6.19)

net : separate network stack (since Linux 2.6.24)

pid : separate PID number space (since Linux 2.6.24)

user : separate user and group IDs (since Linux 2.6.23, removed in 3.8)

cgroup : separate control groups (since Linux 4.6)

Each process has namespace entries visible under

/proc/PID/ns

:

<code>[root@host ns]# ls -l /proc/1/ns
lrwxrwxrwx 1 root root 0 Sep 23 19:53 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 root root 0 Sep 23 19:53 mnt -> mnt:[4026531840]
lrwxrwxrwx 1 root root 0 Sep 23 19:53 net -> net:[4026531956]
lrwxrwxrwx 1 root root 0 Sep 23 19:53 pid -> pid:[4026531836]
lrwxrwxrwx 1 root root 0 Sep 23 19:53 user -> user:[4026531837]
lrwxrwxrwx 1 root root 0 Sep 23 19:53 uts -> uts:[4026531838]
</code>

clone

The

clone

system call creates a new process and can place it in new namespaces using flags.

<code>#define _GNU_SOURCE
#include <sched.h>

int clone(int (*fn)(void *), void *child_stack,
          int flags, void *arg, ... /* pid_t *ptid, void *newtls, pid_t *ctid */ );
</code>

Relevant flags:

CLONE_NEWCGROUP – cgroup namespace

CLONE_NEWIPC – IPC namespace

CLONE_NEWNET – network namespace

CLONE_NEWNS – mount namespace

CLONE_NEWPID – PID namespace

CLONE_NEWUSER – user namespace

CLONE_NEWUTS – UTS namespace

<code>pid = clone(childFunc, stackTop, CLONE_NEWUTS | SIGCHLD, argv[1]);
</code>

setns

The

setns

system call lets a thread join an existing namespace using a file descriptor from

/proc/PID/ns/

.

<code>#define _GNU_SOURCE
#include <sched.h>

int setns(int fd, int nstype);

/* fd is a file descriptor to a namespace file; nstype 0 means any namespace */
</code>

Typical usage:

Call

setns

to specify the thread's namespace.

Call

execvp

to execute the desired program inside that namespace.

nsenter

nsenter wraps setns, allowing the user to specify a target PID instead of a file descriptor. It locates the appropriate

/proc/PID/ns/FD

files and runs the given program inside the selected namespaces.

References

Container network debugging: https://tencentcloudcontainerteam.github.io/tke-handbook/skill/capture-packets-in-container.html

man page: nsenter – http://www.man7.org/linux/man-pages/man1/nsenter.1.html

man page: clone – http://www.man7.org/linux/man-pages/man2/clone.2.html

man page: setns – http://www.man7.org/linux/man-pages/man2/setns.2.html