Efficient Linux System Auditing: Tools, Filters, and Best Practices

When performing security audits or troubleshooting, recording host system actions—such as adding users, renaming files, or executing commands—can be valuable, but excessive logs increase analysis cost, especially when many hosts send data to a remote server.

Most system actions are redundant (e.g., cron jobs, trusted daemons) and generate large amounts of log data that are rarely useful for audit analysis. Therefore, filtering rules should be applied to avoid recording unnecessary information, such as repetitive cron operations, and to prevent logging sensitive data like password‑containing commands.

Ignore records generated by cron and daemon processes.

Ignore sensitive command‑line or script operations that contain passwords.

Ignore records from monitoring users (e.g., nagios, zabbix, prometheus).

Ignore operations that generate logs at high frequency.

The second point is optional; when transmitting logs in clear text to a remote server, it is recommended to omit them. The fourth point deserves emphasis: logging every connect and accept system call on a web host can produce massive logs, burdening the kernel and network transmission, so such exhaustive auditing should only be enabled when needed.

The following methods can be used to audit system operations:

History recording

Custom bash recording

Snoopy recording

auditd recording

eBPF recording

History Recording

The traditional history method sends command history to syslog. It is simple but has major drawbacks for auditing:

Easily modified or bypassed.

Lacks context information (pid, uid, sid, etc.).

Cannot record operations inside shell scripts.

Cannot record non‑login actions.

Difficult to implement filtering rules.

Custom Bash Recording

Custom bash modifies the bash source to add audit logs, allowing additional context, but it shares many limitations with the history method: easy to bypass (users can switch to csh, zsh), cannot record script‑internal operations, filtering may be limited, and maintaining patched bash versions is labor‑intensive.

Snoopy Recording

Snoopy wraps the execv / execve system calls via a preload library, recording every command execution regardless of tty presence. Advantages include:

Hard to bypass once PRELOAD is set.

Records detailed process context even without a tty.

Captures commands executed inside shell scripts.

Logs command arguments, including usernames and passwords.

Rich filtering rules (exclude specific daemons, uids, etc.).

Example log entry:

Oct 27 11:34:31 cz-t1 snoopy[24814]: [time_ms:778 login:cz uid:0 pid:24814 ppid:24676 sid:24579 tty:/dev/pts/0 cwd:/root filename:/bin/uptime username:root]: uptime -p

Drawbacks:

Only supports execv/execve‑related calls.

Without filtering, log volume can be high.

Does not currently filter sensitive information.

Typical filtering configuration to ignore cron, a custom daemon, and the zabbix user:

# zabbix uid is 992</code><code>filter_chain = exclude_uid:992;exclude_spawns_of:crond,my-daemon

Additional exclude_comm rule can ignore specific commands (e.g., mysql, mongo, redis‑cli) that often contain passwords:

filter_chain = exclude_uid:992;exclude_comm:mysql,mongo,redis-cli

Never and always rules have different supported -F fields; ignoring by executable path requires a never rule, which is less convenient than snoopy’s exclude_comm .

auditd Recording

auditd leverages kernel‑level support (kauditd) to provide a comprehensive framework capable of monitoring virtually any event, surpassing snoopy’s scope.

Generated logs include full process context and arguments, for example:

type=SYSCALL msg=audit(1603800704.305:5304075): arch=c000003e syscall=59 success=yes exit=0 a0=1c79fd0 a1=1bf51a0 a2=1bd4450 a3=7ffe7270d320 items=2 ppid=95264 pid=99702 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=571973 comm="mysql" exe="/usr/bin/mysql" key="command"</code><code>type=EXECVE msg=audit(1603800704.305:5304075): argc=5 a0="/usr/bin/mysql" a1="-h" a2="127.0.0.1" a3="-P" a4="3301"

auditd’s architecture separates policy control (auditctl) from event collection (auditd) and distribution (audisp). Policies are defined with -a or -w and stored in /etc/audit/rules.d/audit.rules. Example rules to ignore common tools:

# ignore common tools</code><code>-a never,exit -F arch=b64 -F exe=/usr/bin/redis-cli</code><code>-a never,exit -F arch=b64 -F exe=/usr/bin/mysql</code><code>-a never,exit -F arch=b64 -F exe=/usr/bin/mongo

eBPF Recording

eBPF, available in newer Linux kernels, enables dynamic tracing. Tools like bpftrace and bcc (e.g., execsnoop) can record all execv / execve executions:

# ./execsnoop</code><code>PCOMM            PID    PPID   RET ARGS</code><code>bash             32647  32302    0 /bin/bash</code><code>id               32649  32648    0 /usr/bin/id -un</code><code>hostname         32651  32650    0 /usr/bin/hostname</code><code>uptime           410    32744    0 /bin/uptime

eBPF requires Linux 4.1+; for full feature support, kernels 4.10 or newer are recommended (e.g., CentOS 8, Debian 10). Older distributions like CentOS 7 support a subset of eBPF starting from version 7.6.

Conclusion

System operation auditing facilitates traceability and troubleshooting, and audit logs can serve as forensic evidence. Security‑sensitive organizations may adopt auditd for comprehensive auditing, while snoopy offers a lightweight alternative. For fine‑grained tracing, eBPF is suitable, but careful filtering is essential to avoid overwhelming log volumes.