Effective Linux System Auditing: Methods, Filters & Best Practices

Often for security audits or troubleshooting we record host system actions such as creating users, renaming files, or executing commands. Detailed logs help audit and debugging, but excessive records increase analysis cost, especially when many hosts send logs to a remote server.

Most system actions are redundant—cron jobs, trusted daemons, etc.—which generate large amounts of logs rarely used for audit. Therefore, when auditing system operations we should add filtering rules to avoid unnecessary information, such as duplicate cron actions, and also avoid logging sensitive data like password‑containing commands.

Ignore records generated by cron or daemon processes.

Ignore records of commands or scripts that contain passwords.

Ignore records from monitoring users (e.g., nagios, zabbix, prometheus).

Ignore frequently generated log operations.

The second point is optional; when transmitting logs in plaintext to a remote server, we recommend ignoring them. The fourth point deserves emphasis: logging every network system call (connect, accept) on a web host can produce massive logs and pressure the kernel and log transport, so such detailed auditing should be enabled only when needed.

Below are the main ways to implement system operation auditing:

history recording

custom bash recording

snoopy recording

auditd recording

eBPF recording

history recording

The traditional history method simply forwards command history to syslog. It is easy to modify, lacks context (pid, uid, sid), cannot record script‑internal operations, non‑login actions, and makes filtering difficult.

custom bash recording

Custom bash adds audit logging to the bash source, allowing extra context, but it shares many drawbacks with history: easy to bypass with other shells, cannot record script actions, limited filtering, and requires frequent bash updates.

snoopy recording

Snoopy wraps execv/execve via preload, recording all command executions with full process context, even from scripts, and supports rich filtering (exclude daemons, specific uids, etc.). Example log:

Oct 27 11:34:31 cz-t1 snoopy[24814]: [time_ms:778 login:cz uid:0 pid:24814 ppid:24676 sid:24579 tty:/dev/pts/0 cwd:/root filename:/bin/uptime username:root]: uptime -p

Drawbacks: only supports execv/execve, can generate excessive logs without rules, and currently lacks sensitive‑information filtering.

Typical snoopy filter configuration to ignore cron, a daemon, and the zabbix user:

# zabbix uid is 992
filter_chain = exclude_uid:992;exclude_spawns_of:crond,my-daemon

Additional exclude_comm rule can ignore specific commands (e.g., mysql, mongo, redis-cli) that often contain passwords:

filter_chain = exclude_uid:992;exclude_comm:mysql,mongo,redis-cli

auditd recording

auditd leverages kernel‑level support (kauditd) to provide a comprehensive framework that can monitor virtually any event, offering detailed process and argument information. Logs are easier to read, and the architecture separates policy control (auditctl) from log collection (auditd, audisp).

Example audit log entry:

type=SYSCALL msg=audit(1603800704.305:5304075): arch=c000003e syscall=59 success=yes exit=0 a0=1c79fd0 a1=1bf51a0 a2=1bd4450 a3=7ffe7270d320 items=2 ppid=95264 pid=99702 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=571973 comm="mysql" exe="/usr/bin/mysql" key="command"
type=EXECVE msg=audit(1603800704.305:5304075): argc=5 a0="/usr/bin/mysql" a1="-h" a2="127.0.0.1" a3="-P" a4="3301"

Audit rules are defined with -a or -w and stored in /etc/audit/rules.d/audit.rules. Example to ignore common tools:

# ignore common tools
-a never,exit -F arch=b64 -F exe=/usr/bin/redis-cli
-a never,exit -F arch=b64 -F exe=/usr/bin/mysql
-a never,exit -F arch=b64 -F exe=/usr/bin/mongo

eBPF recording

eBPF, available in Linux 4.1+, enables dynamic tracing. Tools like bpftrace and bcc (e.g., execsnoop) can capture all execv/execve calls with minimal overhead. Example output of execsnoop:

# ./execsnoop
PCOMM            PID    PPID   RET ARGS
bash             32647  32302    0 /bin/bash
id               32649  32648    0 /usr/bin/id -un
hostname         32651  32650    0 /usr/bin/hostname
uptime           410    32744    0 /bin/uptime

eBPF requires Linux 4.1+ (full support from 4.10) and works best on newer distributions such as CentOS 8 or Debian 10.

Conclusion

System operation auditing facilitates issue tracing and can serve as forensic evidence; security‑sensitive organizations may adopt auditd for comprehensive auditing, while snoopy offers a lightweight alternative, and eBPF provides fine‑grained tracing. In practice, combine appropriate filtering rules and, if needed, forward logs to platforms like ELK while avoiding excessive, low‑value data.