Master Linux Log Management: Crontab Cleanup and Logrotate Best Practices

Log Management Goals

Effective log management consists of two main parts: (1) defining useful log content—such as log anchors and format—to aid troubleshooting, and (2) establishing archiving rules, including rotation method (by date, size, etc.) and retention period (e.g., keep only the last month).

Special Log Scenarios

Some services generate uncontrolled log files unless explicitly configured:

Applications started with nohup without log redirection write continuously to nohup.out or a single redirected file.

MySQL allows setting a log file path but does not automatically clean old logs.

nginx lets you set log file paths (default access.log, error.log) but also lacks automatic cleanup.

In these cases, logs grow indefinitely, eventually exhausting disk space and causing system failures.

Special Tool – Scheduled Cleanup

You can use Linux’s crontab together with a cleanup script to delete logs older than a certain period. Example:

crontab -e</code>
<code># Delete logs older than 7 days</code>
<code>1 0 * * * find /logs.dir/ -mtime +7 | xargs rm -rf

Common Pitfalls with Log Files

Renaming a log file and creating a new one does not stop the running process from writing to the original file.

Removing a log file with rm -f requires restarting the service; otherwise the file handle remains and disk space is not freed.

Deleted but still‑open files are invisible to ls and du, yet df shows the real usage; you must use lsof to locate them.

Recommended Actions

For one‑time cleanup, truncate the file (e.g., echo > log.log) instead of deleting it.

If a file was already removed, run lsof | grep -i deleted to find the holding process and restart it.

To keep logs while limiting storage, use logrotate with a copy‑and‑truncate strategy: copy the current log to an archive and then truncate the original file.

Special Tool – logrotate

logrotate

is a Linux‑integrated log management utility scheduled via cron. It can define rotation cycles, compression, retention count, and more, but it can only rotate logs that the application writes to.

Commonly used parameters include:

Rotation period (daily, weekly, monthly)

File extension

Rotation method (copytruncate, create, etc.)

Compression options

Retention count

Example nginx configuration ( /etc/logrotate.d/nginx):

logrotate -d /etc/logrotate.d/nginx</code>
<code>vi /etc/logrotate.d/nginx</code>
<code>/usr/share/nginx/log/*.log {</code>
<code>    daily</code>
<code>    missingok</code>
<code>    rotate 7</code>
<code>    compress</code>
<code>    delaycompress</code>
<code>    notifempty</code>
<code>    create 644 root root</code>
<code>    sharedscripts</code>
<code>    postrotate</code>
<code>        [ ! -f /var/run/nginx.pid ] || kill -USR1 `cat /var/run/nginx.pid`</code>
<code>    endscript</code>
<code>}

Schedule it with cron:

echo "0 0 * * * /usr/sbin/logrotate -vf /etc/logrotate.d/nginx > /dev/null 2>&1" >> /var/spool/cron/root

Key options explained: compress / nocompress: gzip the rotated logs or not. copytruncate: copy the current log and truncate it, useful for files still open by a process. create: specify permissions and ownership for the new log file. delaycompress / nodelaycompress: defer compression to the next rotation. missingok: ignore missing files. notifempty / ifempty: rotate only if the file is non‑empty or always. oldir / noolddir: place rotated files in a separate directory. size / minsize: rotate only when the file reaches a certain size (e.g., size 100M).

Appendix – Simple logrotate Configurations

# MySQL logs</code>
<code>/data/mysql/log/mysqld.log {</code>
<code>    daily</code>
<code>    dateext</code>
<code>    copytruncate</code>
<code>    notifempty</code>
<code>    missingok</code>
<code>    olddir backup</code>
<code>    rotate 60</code>
<code>    compress</code>
<code>}</code>

<code># nginx logs</code>
<code>/usr/local/nginx/logs/access.log</code>
<code>/usr/local/nginx/logs/error.log {</code>
<code>    daily</code>
<code>    dateext</code>
<code>    copytruncate</code>
<code>    notifempty</code>
<code>    missingok</code>
<code>    olddir backup</code>
<code>    rotate 30</code>
<code>    compress</code>
<code>}

Appendix – Log Management for Common Components

nginx: no automatic cleanup; logs are written to a single file continuously.

MySQL: no automatic cleanup; logs grow without rotation.

Zookeeper: supports automatic cleanup via size/count limits; configured via log4j.

Redis: does not auto‑clean; only minimal core logs are recorded.

Kafka data logs (topic, offset): auto‑cleanup configurable; managed via Kafka configs.

Kafka operation logs: stored in logs/, support rotation but not automatic deletion; configured via log4j.