How to Efficiently Audit Linux System Operations Without Overloading Logs

When auditing Linux systems for security or troubleshooting, detailed operation logs are valuable, but excessive or redundant records can overwhelm analysis, especially when logs from many hosts are sent to a remote server.

Ignore records generated by cron jobs and daemons.

Ignore sensitive commands that contain passwords.

Ignore logs from monitoring users such as nagios, zabbix, or prometheus.

Ignore high‑frequency log‑generating operations.

Below are the main ways to implement system operation auditing.

history method

The traditional

history

approach simply forwards command history to syslog, but it has major drawbacks:

Easily modified or bypassed.

Lacks context information (pid, uid, sid, etc.).

Cannot record operations inside shell scripts.

Cannot capture non‑login activities.

Difficult to apply filtering rules.

Custom bash method

Customizing the Bash source adds audit logging with richer context, yet it shares many limitations with the history method and adds maintenance overhead:

Can be bypassed by using other shells (csh, zsh, …).

Cannot record operations inside shell scripts.

Filtering rules are often simplistic.

Requires frequent Bash updates to stay effective.

snoopy method

Snoopy wraps the

execv

/

execve

system calls via a preload library, recording every command execution with full process context. Advantages include:

Hard to bypass once

PRELOAD

is set.

Records exec calls regardless of tty presence, including detailed arguments.

Captures commands executed inside shell scripts.

Logs command parameters such as usernames and passwords.

Rich filtering rules allow exclusion by daemon, uid, or specific commands.

Example log entry:

<code>Oct 27 11:34:31 cz-t1 snoopy[24814]: [time_ms:778 login:cz uid:0 pid:24814 ppid:24676 sid:24579 tty:/dev/pts/0 cwd:/root filename:/bin/uptime username:root]: uptime -p</code>

Drawbacks of snoopy:

Only supports execv/execve‑related calls.

Without proper filters, log volume can become large.

Does not currently filter sensitive information.

Sample snoopy filter configuration to ignore cron, a custom daemon, and the zabbix user:

<code># zabbix uid is 992
filter_chain = exclude_uid:992;exclude_spawns_of:crond,my-daemon</code>

Additional rule to exclude specific commands (e.g., mysql, mongo, redis‑cli) that often contain passwords:

<code>filter_chain = exclude_uid:992;exclude_comm:mysql,mongo,redis-cli</code>

auditd method

Auditd leverages kernel‑level support (kauditd) to provide a comprehensive framework capable of monitoring virtually any event, offering richer context than snoopy.

Typical auditd log example:

<code>type=SYSCALL msg=audit(1603800704.305:5304075): arch=c000003e syscall=59 success=yes exit=0 a0=1c79fd0 a1=1bf51a0 a2=1bd4450 a3=7ffe7270d320 items=2 ppid=95264 pid=99702 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=571973 comm="mysql" exe="/usr/bin/mysql" key="command"
type=EXECVE msg=audit(1603800704.305:5304075): argc=5 a0="/usr/bin/mysql" a1="-h" a2="127.0.0.1" a3="-P" a4="3301"</code>

Auditd rules are managed with

auditctl

and can be stored in

/etc/audit/rules.d/audit.rules

. Example to ignore common tools:

<code>### ignore common tools
-a never,exit -F arch=b64 -F exe=/usr/bin/redis-cli
-a never,exit -F arch=b64 -F exe=/usr/bin/mysql
-a never,exit -F arch=b64 -F exe=/usr/bin/mongo

# Kernel module loading/unloading
-a always,exit -F perm=x -F auid!=-1 -F path=/sbin/insmod -k modules</code>

Note: “never” rules support a limited set of -F fields; excluding by executable path must use “never”, which is less convenient than snoopy’s exclude_comm .

eBPF method

eBPF, available in modern Linux kernels (4.1+), enables dynamic tracing. Tools like

bpftrace

and

bcc

(e.g.,

execsnoop

) can record all

execv

/

execve

calls.

<code># ./execsnoop
PCOMM            PID    PPID   RET ARGS
bash             32647  32302    0 /bin/bash
id               32649  32648    0 /usr/bin/id -un
hostname         32651  32650    0 /usr/bin/hostname
uptime           410    32744    0 /bin/uptime</code>

eBPF requires Linux 4.1+ (full support from 4.10). Distributions such as CentOS 8 or Debian 10 are recommended for production use.

Summary

System operation auditing helps trace and diagnose issues while providing forensic evidence for security‑sensitive environments. For most use cases, combining snoopy or auditd with carefully crafted filtering rules offers a balanced solution, and eBPF can be employed for deeper, low‑overhead tracing when needed.