Master Log Management: Strategies, Tools, and Logrotate Configuration

Log Management Goals

Log management generally consists of two parts: (1) log content – well‑structured logs (anchors, format, etc.) provide valuable execution records and aid troubleshooting; (2) log archiving rules – how logs are split (by date, size, etc.) and retained (e.g., keep only the last month).

Special Log Scenarios

Some services generate uncontrolled log files if not specially configured:

Applications started with nohup without log redirection or redirected to a single file keep writing to nohup.out indefinitely.

MySQL can set a log file path but does not clean logs automatically.

Nginx allows custom log templates and paths (default access.log, error.log) but also lacks automatic cleanup.

In these cases logs grow without bound, eventually exhausting disk space and causing system failures.

Special Tool – Scheduled Cleanup

Use Linux crontab together with a cleanup script, for example:

crontab -e
# Keep logs from the last 7 days
1 0 * * * find /logs.dir/ -mtime +7 | xargs rm -rf

Log File Handling Pitfalls

When a component keeps a file handle open, the following issues arise:

Renaming a log file while the process continues writing to the old file.

Deleting a log file ( rm -f) requires restarting the service; otherwise the space is not freed.

Deleted but still‑open files are invisible to ls or du, yet they occupy space shown by df -h. Use lsof to locate them.

Solutions:

For one‑time cleanup, truncate the file (e.g., echo > log.log) instead of deleting it.

If the file was already removed, find the owning process with lsof | grep -i deleted and restart it.

To retain log data while limiting size, use logrotate with a copy‑and‑truncate strategy.

Special Tool – logrotate

logrotate

is a Linux‑integrated log management utility scheduled via crontab. It defines storage rules for log files, though it can only rotate logs that the application writes to.

Commonly used options include:

Rotation period (daily, weekly, monthly)

File extension

Rotation method (copytruncate, create, etc.)

Compression

Number of retained archives

Example crontab entry to run logrotate daily:

0 0 * * * /usr/sbin/logrotate -vf /etc/logrotate.d/nginx > /dev/null 2>&1

Key logrotate options (brief description): compress / nocompress – gzip the rotated logs. copytruncate – copy the current log and truncate the original (useful for open files). missingok – ignore missing logs. notifempty – skip rotation if the log is empty. rotate N – keep N rotated copies. size – rotate when the log reaches a given size (e.g., size 100M).

Appendix: Simple logrotate Configurations

MySQL {
    daily
    dateext
    copytruncate
    notifempty
    missingok
    olddir backup
    rotate 60
    compress
}

nginx {
    daily
    dateext
    copytruncate
    notifempty
    missingok
    olddir backup
    rotate 30
    compress
}

Appendix: Log Management for Common Components

nginx – no automatic cleanup; logs continuously append to a single file.

MySQL – same limitation as nginx.

Zookeeper – supports automatic cleanup via size and count limits; configured via log4j.

Redis – records only minimal core logs; usually no cleanup needed.

Kafka data logs (topic, offset) – automatic cleanup configurable.

Kafka operation logs – stored in the installation logs directory; supports rotation but not automatic cleanup; managed via log4j.