Mastering Crontab: From Basics to Production‑Ready Scheduling

Overview

crontab is the default Linux scheduler. While simple, it often causes incidents such as missing environment variables, duplicate executions, and unredirected output that fills the mail queue. The article shows a real failure where a backup script without file locking ran twice, corrupting data, and then presents production‑grade practices: lock every job, redirect all output, and log execution.

Key Features

Based on the crond daemon : crond wakes up every minute, reads all crontab files and launches due jobs. Memory usage is ~2 MB and CPU is near zero.

User‑level and system‑level configurations : user crontabs are edited with crontab -e and stored in /var/spool/cron/USERNAME (CentOS) or /var/spool/cron/crontabs/USERNAME (Ubuntu). System crontabs live in /etc/crontab and /etc/cron.d/ and include a username column.

Flexible time expressions : five fields (minute hour day month weekday) support *, ,, -, /. Common patterns (every 5 minutes, weekdays, specific dates, quarterly) are covered. Complex “last weekday of the month” logic is better handled with systemd timers.

Typical Use Cases

Periodic maintenance (log cleanup, backup, certificate renewal) – low precision, long intervals.

Regular data processing (pulling data, generating reports, syncing files) – ensure execution time is less than the interval to avoid overlap.

Monitoring and inspection (service health checks, disk usage) – high frequency (1‑5 min) with lightweight scripts.

Environment Requirements

OS: CentOS 6+/RHEL 6+/Ubuntu 16.04+ (all include cron).

cronie package version ≥ 1.4.11 (CentOS 7 default).

flock utility from util‑linux ≥ 2.23 (file‑lock support).

systemd version ≥ 219 (CentOS 7) or ≥ 229 (CentOS 8) for timer features.

Mail service (postfix or sendmail) optional – cron sends output via email if not redirected.

Preparation

# Verify cron package and service
rpm -qa | grep cronie          # CentOS/RHEL
systemctl status crond

dpkg -l | grep cron           # Ubuntu/Debian
systemctl status cron

# List current crontab
crontab -l

# Check recent cron logs
# CentOS
tail -20 /var/log/cron
# Ubuntu
journalctl -u cron --since "10 minutes ago"

Core Configuration

Time‑Expression Details

┌───────────── minute (0‑59)
│ ┌───────────── hour (0‑23)
│ │ ┌───────────── day of month (1‑31)
│ │ │ ┌───────────── month (1‑12)
│ │ │ │ ┌───────────── day of week (0‑7, 0/7 = Sunday)
│ │ │ │ │
* * * * * command

Special characters: * – any value. , – list of values (e.g., 0,30 * * * *). - – range (e.g., 0 9-17 * * *). / – step (e.g., */5 * * * *).

Note: the day‑and‑weekday fields are OR‑combined; 0 2 15 * 5 means “15th of the month OR every Friday at 2 am”.

User‑level crontab

# Edit current user crontab
crontab -e
# List current user crontab
crontab -l
# Remove all entries (dangerous)
crontab -r
# Import from file
crontab /path/to/file.crontab
# Edit another user (root required)
sudo crontab -u nginx -e

Crontab files are stored in /var/spool/cron/USERNAME (CentOS) or /var/spool/cron/crontabs/USERNAME (Ubuntu). Always use crontab -e to benefit from syntax checking.

System‑level crontab

# /etc/crontab – system crontab (includes username column)
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

Additional snippets can be placed in /etc/cron.d/. Files in /etc/cron.d/ must not contain a dot in the filename, otherwise run-parts skips them.

Environment‑Variable Pitfalls

cron runs with a minimal environment ( PATH=/usr/bin:/bin on CentOS,

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

on Ubuntu). Missing variables cause most failures.

# Capture cron environment for comparison
* * * * * env > /tmp/cron-env.txt
# Compare with interactive shell
env > /tmp/shell-env.txt
diff /tmp/cron-env.txt /tmp/shell-env.txt

Three ways to fix the issue:

Load the full login environment inside the script (e.g., source /etc/profile && source ~/.bashrc).

Define needed variables in the crontab header (e.g., SHELL=/bin/bash,

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

, MAILTO="").

Use absolute paths for all commands.

Output Redirection

# Correct redirection – send both stdout and stderr to a log file
* * * * * /usr/local/bin/check.sh >> /var/log/cron-jobs/check.log 2>&1
# Suppress all output
* * * * * /usr/local/bin/check.sh > /dev/null 2>&1
# Disable email notifications globally
MAILTO=""

Always redirect output; otherwise the mail queue will grow quickly.

Prevent Duplicate Execution with flock

# One‑liner in crontab (every 5 min)
*/5 * * * * flock -xn /tmp/backup.lock -c '/usr/local/bin/backup.sh >> /var/log/cron-jobs/backup.log 2>&1'

# Wrapper script example
#!/bin/bash
LOCK_FILE="/tmp/${0##*/}.lock"
exec 200>"$LOCK_FILE"
flock -xn 200 || { echo "SKIP: another instance running"; exit 0; }
# ...rest of script...

Place lock files in /tmp; avoid /var/lock on systems that clean it.

systemd‑timer Alternative

systemd timers provide second‑level precision, automatic journald logging, resource limits, missed‑run compensation, and random delay.

# /etc/systemd/system/backup.service
[Unit]
Description=Daily backup task

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
User=root
MemoryMax=512M
CPUQuota=50%
TimeoutStartSec=3600

# /etc/systemd/system/backup.timer
[Unit]
Description=Run backup daily at 2:00 AM

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=300
AccuracySec=1s

[Install]
WantedBy=timers.target

Enable and start the timer:

sudo systemctl daemon-reload
sudo systemctl enable backup.timer
sudo systemctl start backup.timer
# Verify status
systemctl status backup.timer
journalctl -u backup.service --since "1 hour ago"

anacron for Missed Runs

# /etc/anacrontab example
1   5   cron.daily   nice run-parts /etc/cron.daily
7   25  cron.weekly  nice run-parts /etc/cron.weekly
@monthly 45 cron.monthly nice run-parts /etc/cron.monthly

anacron stores timestamps in /var/spool/anacron/ and runs jobs that were missed while the machine was powered off.

Best Practices

Performance Optimisation

Stagger heavy jobs : avoid running backup, log‑archive, and data sync all at 02:00 am. Spread them to 02:00, 03:00, 04:00 to prevent I/O saturation.

Keep high‑frequency scripts lightweight : aim for execution time < ⅓ of the interval; replace external commands with direct /proc reads where possible.

Use nice and ionice for non‑critical jobs:

# nice -n 19 ionice -c 3 /usr/local/bin/backup.sh >> /var/log/cron-jobs/backup.log 2>&1

Security Hardening

Restrict cron usage via /etc/cron.allow (list allowed users) or /etc/cron.deny (list denied users). Set permissions to 600.

Ensure script files are owned by root (or the service account) and have mode 750.

Never embed passwords or API keys directly in crontab lines; store them in protected config files (e.g., /root/.my.cnf with mode 600) and source them.

High Availability & Backup

Export all crontabs daily and store under version control:

for user in $(cut -d: -f1 /etc/passwd); do
  crontab -u "$user" -l > /var/backup/crontab/${user}.crontab 2>/dev/null || true
done
cp /etc/crontab /var/backup/crontab/etc-crontab
cp -r /etc/cron.d/ /var/backup/crontab/cron.d/
cd /var/backup/crontab && git init && git add -A && git commit -m "crontab backup $(date +%Y%m%d)"

For critical jobs, deploy on primary and standby nodes with a simple VIP‑based leader election or a distributed lock (Redis/etcd) to avoid double execution.

Audit crontab changes by wrapping the crontab binary (e.g., in /etc/profile.d/crontab-audit.sh) to log user, action, and PID.

Common Pitfalls

%

is special in crontab (represents a newline). Escape it as \% in commands, e.g., date +\%Y\%m\%d. crontab -r deletes all entries without confirmation – always back up first.

Cron runs with HOME as the working directory; use absolute paths in scripts.

Changes to /etc/crontab or files in /etc/cron.d/ may take a few minutes to be noticed by older crond versions.

Troubleshooting & Monitoring

Log Inspection

# CentOS – cron log
tail -f /var/log/cron
# Ubuntu – filter syslog
grep CRON /var/log/syslog | tail -20
# Unified view via journalctl
journalctl -u crond --since "1 hour ago"
journalctl -u cron --since "1 hour ago"

Typical Problems & Debug Steps

Verify the cron service is active ( systemctl status crond or systemctl status cron).

Confirm the crontab entry exists ( crontab -l).

Check logs for execution records.

Ensure the user is not blocked by /etc/cron.allow / /etc/cron.deny.

Validate the time expression with an online tool (e.g., crontab.guru).

Test script permissions ( chmod +x).

Environment‑Diff Debugging

# Temporary task to capture cron environment
* * * * * env > /tmp/cron-env.txt 2>&1
# Compare with interactive environment
env > /tmp/shell-env.txt
diff /tmp/cron-env.txt /tmp/shell-env.txt

Performance Metrics

# Check crond process status and resource usage
systemctl is-active crond   # or cron on Ubuntu
ps aux | grep crond
# Count today's cron executions
grep "$(date +%b\ %d)" /var/log/cron | wc -l
# Detect mail queue size
du -sh /var/spool/mail

Prometheus Alert Rules (excerpt)

# cron daemon down
- alert: CrondDown
  expr: node_systemd_unit_state{name="crond.service",state="active"} != 1
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "crond service not running"
    description: "Host {{ $labels.instance }} has crond stopped; scheduled tasks will not execute."

# Mail queue size too large
- alert: CronMailQueueLarge
  expr: node_directory_size_bytes{directory="/var/spool/mail"} > 104857600
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Cron mail queue exceeds 100 MB"
    description: "Host {{ $labels.instance }} mail queue large – likely missing output redirection."

Custom Metrics Collector (node_exporter textfile)

#!/bin/bash
METRICS="/var/lib/node_exporter/textfile_collector/cron_metrics.prom"
TMP="${METRICS}.tmp"
SUMMARY="/var/log/cron-jobs/execution-summary.csv"
{
  # crond daemon state
  systemctl is-active crond &>/dev/null && echo "cron_daemon_running 1" || echo "cron_daemon_running 0"

  # Number of user crontab entries
  TASK_COUNT=$(crontab -l 2>/dev/null | grep -v '^#' | grep -v '^$' | wc -l)
  echo "cron_task_total $TASK_COUNT"

  # Mail queue size (bytes)
  MAIL_SIZE=$(du -sb /var/spool/mail 2>/dev/null | awk '{print $1}')
  echo "cron_mail_queue_bytes ${MAIL_SIZE:-0}"

  # Export per‑task exit code and duration from the summary CSV (if present)
  if [ -f "$SUMMARY" ]; then
    tail -50 "$SUMMARY" | while IFS=',' read -r exec_time task_name exit_code duration; do
      echo "cron_task_exit_code{task_name=\"$task_name\"} $exit_code"
      echo "cron_task_duration_seconds{task_name=\"$task_name\"} $duration"
    done | sort -u -t'{' -k1,1
  fi
} > "$TMP"
mv "$TMP" "$METRICS"

Backup & Restore

# Backup script (run daily via cron)
#!/bin/bash
set -euo pipefail
BACKUP_DIR="/var/backup/crontab"
DATE=$(date +%Y%m%d)
DEST="$BACKUP_DIR/$DATE"
mkdir -p "$DEST"
# Export user crontabs
for user in $(cut -d: -f1 /etc/passwd); do
  crontab -u "$user" -l 2>/dev/null && crontab -u "$user" -l > "$DEST/${user}.crontab" || true
done
# System crontabs
cp /etc/crontab "$DEST/etc-crontab"
cp -r /etc/cron.d/ "$DEST/cron.d/"
cp /etc/anacrontab "$DEST/" 2>/dev/null || true
# Cleanup older backups (>30 days)
find "$BACKUP_DIR" -maxdepth 1 -type d -mtime +30 -exec rm -rf {} \;

Restore steps:

Verify the backup directory exists (e.g., /var/backup/crontab/20240115).

Restore a user crontab: crontab -u root /var/backup/crontab/20240115/root.crontab.

Restore system files: cp /var/backup/crontab/20240115/etc-crontab /etc/crontab and cp -r /var/backup/crontab/20240115/cron.d/* /etc/cron.d/.

Restart the cron service and verify jobs run.

Conclusion

Always redirect stdout and stderr; otherwise mail queues explode.

Environment differences are the top cause of failures – load profiles, define PATH, or use absolute paths.

Use flock (or systemd timers) to guarantee single execution.

Escape % in crontab commands (e.g., date +\%Y\%m\%d).

systemd timers provide second‑level precision, resource limits, and missed‑run compensation; prefer them for complex schedules.

Back up crontabs regularly and keep them under version control.

References

crontab.guru – online crontab expression validator.

cronie project – source and documentation for the CentOS/RHEL cron implementation.

systemd.timer manual – full configuration reference.

flock manual – detailed usage of file locks.