Mastering Sysdig: Unified System Visibility, Filtering, and Container Monitoring

What Is Sysdig?

Sysdig is an open‑source universal system visibility tool with native container support. It provides monitoring, analysis, and troubleshooting capabilities on Linux systems, consolidating the functionality of utilities such as strace, tcpdump, htop, iftop, and lsof.

Key Advantages

Integration : Replaces multiple specialized commands with a unified syntax.

Power : Captures real‑time system data and can store it for later analysis, covering CPU, memory, disk I/O, network I/O, processes, files, and network connections.

Flexibility : Uses a tcpdump‑style filter language and supports custom Lua scripts (chisels) for advanced analysis.

How Sysdig Works

Sysdig registers hooks in a kernel driver module to intercept system calls. When a call occurs, its information is copied to a buffer, then user‑space components parse, filter, and present the data via the sysdig CLI.

Installation

Installation instructions are available in the official documentation. Ensure your kernel version meets Sysdig’s requirements. For container‑based installations, mount host directories into the container, e.g.:

$ docker run -i -t --name sysdig --privileged \
  -v /var/run/docker.sock:/host/var/run/docker.sock \
  -v /dev:/host/dev -v /proc:/host/proc:ro \
  -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro \
  -v /usr:/host/usr:ro sysdig/sysdig

Basic Usage

Run sysdig with root privileges to start capturing events: $ sudo sysdig Press Ctrl+C to stop. The default output format is a space‑separated line containing fields such as event number, timestamp, CPU, process name, thread ID, direction, type, and arguments.

Filtering

Use the syntax sysdig [options] [filter]. Filters can target file descriptors, processes, events, users, syslog entries, etc. Example to capture only TCP accept events: $ sudo sysdig evt.type=accept Supported operators include =, !=, >=, <=, contains, in, and logical and, or, not. Example combining conditions:

$ sysdig "not (fd.name contains /proc or fd.name contains /dev)"

Custom Output Formats

Use -p with a printf‑style string to select fields. Prefix fields with %. Example printing user and directory for chdir events:

$ sysdig -p"user:%user.name dir:%evt.arg.path" evt.type=chdir

Saving to Files

Capture to a file with -w and later read with -r:

$ sudo sysdig -w sysdig-trace.scap
$ sudo sysdig -r sysdig-trace.scap

Control file size with options like -C (max size per file), -W (number of files), -G (time‑based rotation), and -e (event count). Compression is available via -z.

Common Parameters

-A

/ --print-ascii: Print buffers as ASCII. -x / --print-hex: Print buffers as hex. -X / --print-hex-ascii: Print both hex and ASCII. -s size : Set capture buffer size (default 80 bytes). -N: Do not resolve port numbers to names (faster).

Chisels – Pre‑Built Lua Scripts

Chisels are reusable Lua scripts that perform specific analyses. List them with: $ sudo sysdig -cl Examples: httplog: Show all HTTP requests. topprocs_cpu: List processes sorted by CPU usage. spy_file: Trace reads/writes to a specific file. netstat: Display network connections.

Run a chisel with -c or get its usage with -i:

$ sudo sysdig -c topprocs_net
$ sudo sysdig -i spy_file

Container Support

Sysdig natively understands Docker and Kubernetes. Add container metadata to events with -pc and filter by container name or ID:

$ sysdig -pc container.name=zen_knuth
$ sysdig -pc -c topconns container.name=wordpress1

For Kubernetes clusters, specify the API server with -k http://master_ip:8080 and optionally a CA certificate with -K filename.

Csysdig – TUI Interface

Csysdig provides a top/htop‑like textual UI for real‑time monitoring and can also read saved capture files.

Further Reading

Consult the Sysdig User Guide and community resources for deeper knowledge, including comparisons with strace, htop, lsof, tcpdump, and iftop.

References

Sysdig User Guide

Linux Troubleshooting Cheatsheet

DigitalOcean: How To Monitor Your Ubuntu 16.04 System with Sysdig

Sysdig vs DTrace vs Strace

Using Sysdig to Monitor Servers and Docker Containers