Production-Grade Shell Scripts: Error Handling, Logging & Cleanup for Linux Ops

Why Basic Scripts Fail in Production

Beginner Shell scripts package repeated commands for simple automation but lack resilience for production environments. The author identifies three core requirements for production-grade scripts: (1) stability—scripts must not crash on errors; they need retry mechanisms and rollback plans; (2) traceability—every operation must leave a clear audit trail with timestamps and levels; (3) graceful exit—scripts must clean up temporary files, locks, and resources on both normal and abnormal termination.

Technique 1: Error Radar with set -euo pipefail

Place these at the top of the script:

#!/bin/bash
set -e          # Exit immediately if any command returns non-zero
set -u          # Treat unset variables as errors
set -o pipefail # Fail pipeline if any command fails, not just the last
# set -x        # Debug mode: print each command (enable during debugging)

Key explanations: set -e: Stops script on command failure (e.g., failed tar backup), preventing subsequent destructive steps like overwriting good backups. set -u: Catches typos in variable names (e.g., BACKUP_Dir vs BACKUP_DIR) before they cause logic errors. set -o pipefail: Ensures a failure in any part of a pipeline (e.g., cat log.txt | grep error where cat fails) triggers script exit.

Tip: To allow a specific command to fail without stopping the script, append || true, e.g., find $BACKUP_DIR -name "*.tar.gz" -mtime +7 || true.

Technique 2: Professional Logging System

Replace echo with a structured logging function that provides timestamps, levels (INFO/WARNING/ERROR), persistent file storage, and colored console output.

#!/bin/bash
LOG_FILE="/var/log/my_script.log"
SCRIPT_NAME=$(basename "$0")

log_message() {
    local level=$1
    local message=$2
    local timestamp=$(date "+%Y-%m-%d %H:%M:%S")
    # Persist to log file
    echo "[$timestamp] [$level] $SCRIPT_NAME: $message" >> "$LOG_FILE"
    # Colorized console output for interactive runs
    if [ -t 0 ]; then
        case $level in
            "INFO") color="\033[32m" ;;   # green
            "WARNING") color="\033[33m" ;; # yellow
            "ERROR") color="\033[31m" ;;  # red
            *) color="\033[0m" ;;
        esac
        echo -e "${color}[$level] $message\033[0m"
    fi
}

# Usage examples
log_message "INFO" "Script started, preparing system log backup"
log_message "WARNING" "Backup directory has less than 500MB free space"
log_message "ERROR" "Source directory /var/log does not exist, creating..."

Advanced optimizations:

Log rotation: Use logrotate to limit retention (e.g., keep 7 days, rotate at 100MB).

Stderr capture: Add exec 2>> "$LOG_FILE" at script start to redirect all standard error to the log file.

Sensitive data filtering: Strip passwords/keys from log output using sed before writing.

Technique 3: Automatic Cleanup with trap

Scripts interrupted by Ctrl+C, kill, or crashes leave temporary files and lock files that block subsequent runs. trap captures exit signals and runs a cleanup function.

#!/bin/bash
LOG_FILE="/var/log/my_script.log"
SCRIPT_NAME=$(basename "$0")
# log_message function omitted for brevity

cleanup() {
    log_message "INFO" "Starting cleanup..."
    # Remove temporary files
    rm -f /tmp/temp_*.txt /tmp/backup_cache.dat
    # Remove lock file to allow future runs
    rm -f /var/lock/my_script.lock
    # Optional: kill child processes
    # pkill -P $$
    log_message "INFO" "Cleanup complete, script exiting"
}

# Trap EXIT (normal), INT (Ctrl+C), TERM (kill)
trap cleanup EXIT INT TERM

# Example: prevent concurrent execution with lock file
if [ -f /var/lock/my_script.lock ]; then
    log_message "ERROR" "Script already running (lock file detected), exiting"
    exit 1
else
    touch /var/lock/my_script.lock
    log_message "INFO" "Lock file created, starting main logic"
fi
# ... main script logic ...

Note: Keep the cleanup function simple and reliable; avoid commands that could fail and obscure the original error.

Integrated Example: Production-Grade Log Backup Script

The article combines all three techniques into a complete backup_syslog.sh script with the following capabilities:

Error protection via set -euo pipefail and stderr redirection

Full-process logging with timestamps, levels, and script name

Automatic cleanup of temp files and lock files on exit

Disk space check before backup (requires 1GB free)

Retry mechanism (up to 3 attempts with 2-second intervals)

Retention policy (delete backups older than 7 days)

Script structure walkthrough:

Configuration section: Centralized variables for paths, retention days, retry count, required space, lock file.

Log function: As defined in Technique 2.

Cleanup function: Removes lock file and temporary backup archives.

Helper check_disk_space : Uses df -m and awk to verify available space in MB.

Main function:

Sets trap cleanup EXIT INT TERM Checks/creates lock file

Ensures backup directory exists

Runs disk space check

Generates timestamped backup filename

Executes tar -zcf with --exclude in a retry loop (max 3), logging each attempt, deleting incomplete archives on failure, sleeping 2 seconds between retries

On success, logs backup size; on total failure, exits with error

Cleans old backups with find -mtime +7 -delete -print | wc -l and logs count

Entry point: Logs start, calls main, logs completion.

#!/bin/bash
set -euo pipefail
exec 2>> "$LOG_FILE"

BACKUP_DIR="/backup/logs"
LOG_DIR="/var/log"
LOG_FILE="/var/log/backup_script.log"
RETENTION_DAYS=7
MAX_RETRIES=3
REQUIRED_SPACE=1024
LOCK_FILE="/var/lock/backup_script.lock"

log_message() { ... }
cleanup() { ... }
check_disk_space() { ... }
main() {
    trap cleanup EXIT INT TERM
    # lock, dir, disk, backup with retry, cleanup old
}
log_message "INFO" "=== System log backup script started ==="
main
log_message "INFO" "=== System log backup script finished ==="

Usage:

Make executable: chmod +x backup_syslog.sh Test with debug: uncomment set -x and run ./backup_syslog.sh Schedule via cron: 0 3 * * * /path/to/backup_syslog.sh Monitor logs:

tail -f /var/log/backup_script.log

Debugging Techniques

Line-by-line tracing: Run bash -x script.sh to print every command and variable value (like a recording).

Static analysis: Install shellcheck ( apt/yum/brew install shellcheck) and run shellcheck script.sh to catch syntax errors, undefined variables, quoting issues.

Local debugging: Wrap suspect code blocks with set -x (enable) and set +x (disable) to limit verbose output to a specific section.

Further Learning Path

Shell security: Prevent command injection, handle secrets safely, apply least privilege (avoid root).

External integration: Use curl for APIs, jq for JSON parsing, getopts / getopt for argument parsing.

Script engineering: Integrate into CI/CD pipelines (Jenkins, GitLab CI) for version control, automated testing, and large-scale deployment.

Good Shell scripts evolve through continuous iteration—adding error handling, logging, and cleanup as real problems surface. Start applying these three pillars to every script today; in three months your automation will run reliably in production, and you will have grown from a script writer into an automation operations engineer.