How to Effectively Audit Linux System Operations and Cut Down Noise

When performing security audits or troubleshooting, recording detailed host system actions (such as user creation, file renames, or command execution) can be valuable, but excessive logs from many hosts quickly become costly to analyze. Redundant entries from cron jobs, trusted daemons, or sensitive commands should be filtered.

Filtering Guidelines

Ignore records generated by cron or other daemons.

Ignore command‑line or script operations that contain passwords.

Ignore logs from monitoring users (e.g., nagios, zabbix, prometheus).

Ignore operations that produce logs at a very high frequency.

These rules help reduce noise while preserving useful audit data.

Auditing Methods

history logging

custom bash logging

snoopy logging

auditd logging

eBPF tracing

history logging

The traditional history approach simply forwards command history to syslog. It is easy to set up but has major drawbacks for auditing:

Easy to modify or bypass.

Lacks context information such as PID, UID, SID.

Cannot capture commands executed inside shell scripts.

Cannot record non‑login operations.

Hard to implement fine‑grained filtering rules.

custom bash logging

Modifying the Bash source to emit audit logs adds more context but shares many of the history method’s limitations and introduces additional maintenance challenges.

Can be bypassed by using other shells (csh, zsh, etc.).

Cannot capture operations inside scripts.

Filtering rules tend to be simplistic.

Requires continual updates to match distribution‑provided Bash versions.

snoopy logging

Snoopy intercepts execv and execve via a preload library, recording every command execution with full process context, even from scripts or non‑interactive sessions.

Hard to bypass once PRELOAD is set.

Captures detailed arguments and user information.

Records script‑internal commands because they invoke execv/execve.

Supports rich filtering (exclude specific daemons, UIDs, or commands).

Example log entry:

Oct 27 11:34:31 cz-t1 snoopy[24814]: [time_ms:778 login:cz uid:0 pid:24814 ppid:24676 sid:24579 tty:/dev/pts/0 cwd:/root filename:/bin/uptime username:root]: uptime -p

Drawbacks:

Only records execv/execve related calls.

Without proper filters, log volume can become overwhelming.

Currently does not support filtering of sensitive data within arguments.

Typical filtering configuration to ignore cron, a custom daemon, and the zabbix user:

# zabbix uid is 992
filter_chain = exclude_uid:992;exclude_spawns_of:crond,my-daemon

Additional exclude_comm rule to drop commands of common tools that may expose passwords:

filter_chain = exclude_uid:992;exclude_comm:mysql,mongo,redis-cli

auditd logging

Auditd leverages the kernel’s native audit subsystem (kauditd) to provide a comprehensive framework capable of monitoring virtually any system event, offering richer context than snoopy.

Sample audit log entry:

type=SYSCALL msg=audit(1603800704.305:5304075): arch=c000003e syscall=59 success=yes exit=0 a0=1c79fd0 a1=1bf51a0 a2=1bd4450 a3=7ffe7270d320 items=2 ppid=95264 pid=99702 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=571973 comm="mysql" exe="/usr/bin/mysql" key="command"
type=EXECVE msg=audit(1603800704.305:5304075): argc=5 a0="/usr/bin/mysql" a1="-h" a2="127.0.0.1" a3="-P" a4="3301"

Audit rules are managed with auditctl (or the -a / -w options) and stored in /etc/audit/rules.d/audit.rules. Example rules to ignore common database client tools:

### ignore common tools
-a never,exit -F arch=b64 -F exe=/usr/bin/redis-cli
-a never,exit -F arch=b64 -F exe=/usr/bin/mysql
-a never,exit -F arch=b64 -F exe=/usr/bin/mongo

## Kernel module loading and unloading
-a always,exit -F perm=x -F auid!=-1 -F path=/sbin/insmod -k modules

Note: The never action supports filtering by exe , while always does not, making never necessary for tool‑path exclusions.

eBPF tracing

eBPF, available in Linux 4.1+, enables dynamic tracing without kernel recompilation. Tools such as bcc and bpftrace provide ready‑made scripts like execsnoop to capture every execv/execve call.

Sample execsnoop output:

# ./execsnoop
PCOMM            PID    PPID   RET ARGS
bash             32647  32302    0 /bin/bash
id               32649  32648    0 /usr/bin/id -un
hostname         32651  32650    0 /usr/bin/hostname
uptime           410    32744    0 /bin/uptime

eBPF requires a relatively recent kernel (4.10+ for full feature support). CentOS 8, Debian 10 and newer distributions are recommended; older kernels may lack some capabilities.

Conclusion

Auditing Linux system operations helps trace security incidents and troubleshoot failures, and the generated logs can serve as forensic evidence. For most environments, snoopy or auditd provide the most practical coverage, while eBPF offers deeper, programmable tracing when needed. Regardless of the method, carefully crafted filtering rules are essential to prevent log overload and protect sensitive information.