Why Your Crontab Jobs Fail: 5 Common Mistakes and How to Fix Them

Introduction

"Why does my backup script work manually but not in crontab?" is a question many ops engineers face. Crontab is the most widely used Linux scheduler, but its simplicity hides pitfalls: an invisible space, a missing environment variable, or a wrong path can silently disable a job. Over 60% of crontab failures stem from five recurring mistakes.

Technical Background: How Crontab Works

History and Evolution

Crontab (Cron Table) originated from the Unix cron daemon in 1975. Its design philosophy is "simple, reliable, efficient":

Simple : plain‑text configuration, easy to read and edit.

Reliable : system‑level daemon, starts at boot.

Efficient : minimal resource usage, checks tasks every minute.

Cron Execution Environment

Cron runs in an extremely minimal environment. The following command shows the actual environment variables available to a cron job:

# View real cron environment variables
* * * * * env > /tmp/cron_env.txt

# Compare with normal shell environment
env > /tmp/shell_env.txt

diff /tmp/cron_env.txt /tmp/shell_env.txt

The differences are striking: PATH is reduced to /usr/bin:/bin, SHELL is /bin/sh, and most other variables are missing. This stripped environment is the root cause of many crontab problems.

Cron Time Expression Details

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of week (0 - 7, 0 and 7 are Sunday)
# │ │ │ │ │
# * * * * * command to execute

Special characters : *: any value ,: list (e.g., 1,3,5) -: range (e.g., 1-5) /: step (e.g., */5 every 5 units) @reboot: run at system start @daily: equivalent to

0 0 * * *

Cron Log Mechanism

Log locations differ by distribution:

Ubuntu/Debian : /var/log/syslog (cron messages mixed with system logs)

CentOS/RHEL : /var/log/cron (dedicated cron log)

Typical log queries:

# Ubuntu/Debian
grep CRON /var/log/syslog | tail -20

# CentOS/RHEL
tail -20 /var/log/cron

Log characteristics:

Only records whether cron scheduled the job.

Does not capture command output or errors.

Does not record success/failure status.

Core Content: 5 Most Common Fatal Errors

Error 1 – Missing Environment Variables (Command Not Found)

Risk : ★★★★★ Probability : 70% Typical Symptom : Works manually, fails in crontab.

Real Case

#!/bin/bash
# backup.sh
mysqldump -u root -pPassword123 --all-databases > /backup/mysql_$(date +%Y%m%d).sql
gzip /backup/mysql_$(date +%Y%m%d).sql
aws s3 cp /backup/mysql_$(date +%Y%m%d).sql.gz s3://my-bucket/backups/

Crontab entry: 0 2 * * * /home/ops/backup.sh When executed by cron the script reports:

/home/ops/backup.sh: line 3: mysqldump: command not found
/home/ops/backup.sh: line 5: aws: command not found

Root cause: cron’s PATH lacks /usr/bin and /usr/local/bin, where mysqldump and aws reside.

Solutions:

Use absolute paths in the script.

Set a full PATH at the top of the script.

Define PATH in the crontab file.

Run the script with a login shell ( /bin/bash -l -c …).

#!/bin/bash
# backup.sh – improved
/usr/bin/mysqldump -u root -pPassword123 --all-databases > /backup/mysql_$(date +%Y%m%d).sql
/bin/gzip /backup/mysql_$(date +%Y%m%d).sql
/usr/local/bin/aws s3 cp /backup/mysql_$(date +%Y%m%d).sql.gz s3://my-bucket/backups/

Error 2 – Relative Paths

Risk : ★★★★★ Probability : 65% Typical Symptom : "No such file or directory".

Real Case

#!/bin/bash
# cleanup.sh
cd logs/
find . -name "*.log" -mtime +7 -delete

Crontab entry: 0 3 * * * /home/ops/scripts/cleanup.sh When cron runs, the working directory is /home/ops, so logs/ cannot be found.

Solutions:

Use absolute paths.

Determine the script’s directory at runtime and build paths from it.

Change to the desired directory inside the crontab line.

#!/bin/bash
# cleanup.sh – absolute path version
cd /home/ops/logs/
find . -name "*.log" -mtime +7 -delete

#!/bin/bash
# cleanup.sh – dynamic path version
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
LOG_DIR="$SCRIPT_DIR/../logs"
cd "$LOG_DIR"
find . -name "*.log" -mtime +7 -delete

Error 3 – Output Not Redirected

Risk : ★★★★☆ Probability : 80% Typical Symptom : No visibility of execution or errors.

Real case: a data‑sync script runs every 5 minutes but produces no logs. */5 * * * * /home/ops/sync_data.sh Solution: redirect both stdout and stderr to a log file.

# Append output and errors
*/5 * * * * /home/ops/sync_data.sh >> /var/log/sync_data.log 2>&1

Alternatively, handle logging inside the script with exec and custom functions.

Error 4 – Wrong Time Expression

Risk : ★★★★☆ Probability : 50% Typical Symptom : Task runs at unexpected times.

Common mistakes:

Writing 0 2 * * * when the intention is “every 2 hours”. Correct: 0 */2 * * *.

Confusing day‑of‑month with day‑of‑week (e.g., 0 0 1 * * vs 0 0 * * 1).

Assuming cron supports “last day of month”; it does not.

Use tools like crontab.guru or croniter to test expressions.

# Example with croniter (Python)
from croniter import croniter
from datetime import datetime
cron = croniter('0 */2 * * *', datetime.now())
for i in range(5):
    print(cron.get_next(datetime))

Error 5 – Permission and User Confusion

Risk : ★★★★☆ Probability : 40% Typical Symptom : "Permission denied" or "Operation not permitted".

Typical scenarios:

Mixing user crontab ( crontab -e) with system crontab ( /etc/crontab) and adding a username where it does not belong.

Editing root’s crontab when the script should run as a regular user.

Missing execute permission on the script or its directory.

SELinux/AppArmor blocking execution.

Fixes:

Ensure the correct crontab is edited ( crontab -e -u ops for user ops).

Add execute bits: chmod +x /home/ops/backup.sh and ensure the directory is searchable ( chmod +x /home/ops).

Check SELinux status ( getenforce) and adjust policies if needed.

Practical Debugging Case: A Failing Report Job

Problem Description

0 8 * * 1-5 /home/reports/generate_report.sh

Symptoms: manual run works, cron run produces no report.

Debugging Steps

Verify cron service is running ( systemctl status cron).

Test a simple per‑minute job and check /tmp/cron_test.log.

Inspect cron logs ( grep CRON /var/log/syslog or /var/log/cron).

Redirect script output to a dedicated log file.

Manually execute the script in a simulated cron environment:

env -i HOME=/home/reports SHELL=/bin/sh PATH=/usr/bin:/bin /bin/sh -c '/home/reports/generate_report.sh'

Findings:

Python not found – use absolute path /usr/bin/python3.

Database password missing – read from a config file instead of relying on environment variables.

Output directory absent – create it at runtime.

Final Robust Script

#!/bin/bash
# generate_report.sh – production version
set -euo pipefail
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
OUTPUT_DIR="$SCRIPT_DIR/output"
LOG_DIR="$SCRIPT_DIR/logs"
CONFIG_FILE="$SCRIPT_DIR/config.ini"
mkdir -p "$OUTPUT_DIR" "$LOG_DIR"
LOG_FILE="$LOG_DIR/report_$(date +%Y%m%d).log"
log(){ echo "[$(date +'%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }
log_error(){ echo "[$(date +'%Y-%m-%d %H:%M:%S')] [ERROR] $*" | tee -a "$LOG_FILE" >&2; }
trap 'log_error "Script failed at line $LINENO"' ERR
log "=== Report generation started ==="
# Dependency check
for cmd in python3 mysql; do
  command -v "$cmd" >/dev/null || { log_error "Required command '$cmd' not found"; exit 1; }
done
# Load config
[ -f "$CONFIG_FILE" ] || { log_error "Config file missing"; exit 1; }
source "$CONFIG_FILE"
# Generate report
/usr/bin/python3 "$SCRIPT_DIR/generate.py" --output "$OUTPUT_DIR" --config "$CONFIG_FILE" || { log_error "Report generation failed"; exit 1; }
log "Report generated successfully"
# Notify
mail -s "Daily Report $(date +%Y-%m-%d)" [email protected] <<< "Report created: $OUTPUT_DIR/report_$(date +%Y%m%d).pdf"
log "=== Report generation completed ==="

Best Practices – 10 Golden Rules for Crontab

Always use absolute paths for commands, scripts, and files.

Explicitly set required environment variables (PATH, SHELL, LANG) either in the crontab header or inside the script.

Redirect all output to log files; include timestamps.

Ensure scripts have a proper shebang and executable permission.

Use lock files or flock to prevent overlapping runs.

Record start time, end time, and duration for each run.

Implement error handling and alerting (email, Slack, etc.).

Periodically review and clean up obsolete crontab entries.

Test in a simulated cron environment or with a high‑frequency schedule before production.

Document each job: purpose, owner, modification date, and related runbook.

Summary and Outlook

Key Takeaways

Environment variables are the biggest trap – cron’s minimal environment differs greatly from an interactive shell.

Absolute paths are essential for commands, scripts, and files.

Logging is the lifeline; without logs you cannot see failures.

Always test changes in a safe environment before deploying.

Permissions (file, directory, user) must be verified.

Alternative Scheduling Solutions

For more complex needs consider:

Systemd timers – native to modern Linux, integrated with journalctl, supports dependencies.

Apache Airflow – DAG‑based orchestration for intricate workflows.

Kubernetes CronJob – container‑native scheduling in cloud‑native environments.

Final Advice

Today: audit all existing crontab jobs and add proper logging. This week: add alerting for critical tasks. This month: build documentation and monitoring for the crontab fleet. Continuously: adopt the test‑before‑deploy habit. Remember, crontab never complains – it simply fails silently. Proactive logging, testing, and verification are the only ways to guarantee that your scheduled jobs run as expected.