Mastering nsenter: Debug Container Networks and Namespaces with Ease

nsenter is a command from the util-linux package that runs a specified program inside the namespace of a given process.

Use Cases

A typical use case is entering a container's network namespace to run host tools such as ip address, ping, telnet, or tcpdump, avoiding the need to inspect the container IP via docker inspect. nsenter can also enter other namespaces such as mnt, uts, ipc, pid, and user, and can set a root directory and working directory.

Usage

Syntax:

nsenter [options] [program [arguments]]

options:
  -t, --target pid          Target process PID whose namespace to enter
  -m, --mount[=file]        Enter mount namespace (optional file)
  -u, --uts[=file]          Enter UTS namespace
  -i, --ipc[=file]          Enter IPC namespace
  -n, --net[=file]          Enter network namespace
  -p, --pid[=file]          Enter PID namespace
  -U, --user[=file]         Enter user namespace
  -G, --setgid gid          Set GID for the program
  -S, --setuid uid          Set UID for the program
  -r, --root[=directory]    Set root directory
  -w, --wd[=directory]      Set working directory

If no program is given, the default is $SHELL.

Example:

# Get the PID of an nginx container
[root@host ~]# docker inspect -f {{.State.Pid}} nginx
5645

# Enter the container's network namespace
[root@host ~]# nsenter -n -t5645
[root@host ~]# ip addr
... (output showing interfaces) ...

In Kubernetes, obtain the container ID first:

# Get containerID via yaml output
[root@node1 test]# kubectl get pod test -oyaml | grep containerID
  - containerID: docker://cf0873782d58...

# Or more precisely
[root@node1 test]# kubectl get pod test -o template --template='{{range .status.containerStatuses}}{{.containerID}}{{end}}'
docker://cf0873782d58...

Principle

Namespace

Linux namespaces isolate various resources of processes. Types include:

mount : separate mount points (since Linux 2.4.19)

ipc : separate IPC objects like message queues, shared memory, semaphores (since Linux 2.6.19)

uts : separate hostname and domain name (since Linux 2.6.19)

net : separate network stack (since Linux 2.6.24)

pid : separate PID number space (since Linux 2.6.24)

user : separate user and group IDs (since Linux 2.6.23, removed in 3.8)

cgroup : separate control groups (since Linux 4.6)

Each process has namespace entries visible under /proc/PID/ns:

[root@host ns]# ls -l /proc/1/ns
lrwxrwxrwx 1 root root 0 Sep 23 19:53 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 root root 0 Sep 23 19:53 mnt -> mnt:[4026531840]
lrwxrwxrwx 1 root root 0 Sep 23 19:53 net -> net:[4026531956]
lrwxrwxrwx 1 root root 0 Sep 23 19:53 pid -> pid:[4026531836]
lrwxrwxrwx 1 root root 0 Sep 23 19:53 user -> user:[4026531837]
lrwxrwxrwx 1 root root 0 Sep 23 19:53 uts -> uts:[4026531838]

clone

The clone system call creates a new process and can place it in new namespaces using flags.

#define _GNU_SOURCE
#include <sched.h>

int clone(int (*fn)(void *), void *child_stack,
          int flags, void *arg, ... /* pid_t *ptid, void *newtls, pid_t *ctid */ );

Relevant flags:

CLONE_NEWCGROUP – cgroup namespace

CLONE_NEWIPC – IPC namespace

CLONE_NEWNET – network namespace

CLONE_NEWNS – mount namespace

CLONE_NEWPID – PID namespace

CLONE_NEWUSER – user namespace

CLONE_NEWUTS – UTS namespace

pid = clone(childFunc, stackTop, CLONE_NEWUTS | SIGCHLD, argv[1]);

setns

The setns system call lets a thread join an existing namespace using a file descriptor from /proc/PID/ns/.

#define _GNU_SOURCE
#include <sched.h>

int setns(int fd, int nstype);

/* fd is a file descriptor to a namespace file; nstype 0 means any namespace */

Typical usage:

Call setns to specify the thread's namespace.

Call execvp to execute the desired program inside that namespace.

nsenter

nsenter wraps setns, allowing the user to specify a target PID instead of a file descriptor. It locates the appropriate /proc/PID/ns/FD files and runs the given program inside the selected namespaces.

References

Container network debugging: https://tencentcloudcontainerteam.github.io/tke-handbook/skill/capture-packets-in-container.html

man page: nsenter – http://www.man7.org/linux/man-pages/man1/nsenter.1.html

man page: clone – http://www.man7.org/linux/man-pages/man2/clone.2.html

man page: setns – http://www.man7.org/linux/man-pages/man2/setns.2.html