Midnight Deployment Crash? One-Click Backup & Rollback Scripts for Code and Database

Three Deployment Scenarios, Three Rollback Strategies

The author outlines three common environments and their native rollback approaches:

Cloud servers / H3C with snapshot platforms : Platform-level full-machine backup works but restores the entire system, taking longer.

Kubernetes clusters : Use kubectl rollout undo to switch image versions; containers restart in seconds.

Traditional physical or bare-metal VMs : No platform snapshots or containers — rely on custom scripts that package code and export databases at the application level, restoring only the service without touching the OS.

Core Principles

Backup everything : Code + database, neither omitted.

Version clarity : Every backup carries a timestamp for traceability to any version.

Fast rollback : One-click execution; no manual command sequences on the server.

Directory Structure

/data/
  ├── www/                  # Code directory
  │   └── project_name/
  ├── backups/              # Backup root
  │   └── project_name/
  │       ├── code/         # Code backups
  │       │   ├── 20260623_020001/
  │       │   ├── 20260622_100000/
  │       │   └── ...
  │       └── db/           # Database backups
  │           ├── 20260623_020001.sql.gz
  │           ├── 20260622_100000.sql.gz
  │           └── ...

Each backup creates a timestamped directory (code) or file (database). Rollback selects the matching version.

backup.sh — One-Click Backup (Code + Database)

Usage: ./backup.sh <project_name> [retain_count] (default retain 7).

#!/bin/bash
set -e
PROJECT_NAME="$1"
KEEP_COUNT="${2:-7}"
if [ -z "$PROJECT_NAME" ]; then
  echo "Usage: $0 <project_name> [retain_count]"
  exit 1
fi
# ========== Config (adjust as needed) ==========
CODE_SOURCE="/data/www/${PROJECT_NAME}"
BACKUP_ROOT="/data/backups/${PROJECT_NAME}"
DB_NAME="${PROJECT_NAME}_db"
DB_USER="backup_user"
DB_PASS="your_password"  # Prefer ~/.my.cnf instead of plaintext
# ==============================================
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
CODE_BACKUP_DIR="${BACKUP_ROOT}/code/${TIMESTAMP}"
DB_BACKUP_FILE="${BACKUP_ROOT}/db/${TIMESTAMP}.sql.gz"
mkdir -p "${BACKUP_ROOT}/code"
mkdir -p "${BACKUP_ROOT}/db"
echo "Starting backup for project: $PROJECT_NAME"
# 1. Backup code
if [ -d "$CODE_SOURCE" ]; then
  cp -r "$CODE_SOURCE" "$CODE_BACKUP_DIR"
  echo "Code backup done: $(du -sh $CODE_BACKUP_DIR | awk '{print $1}')"
else
  echo "Code directory not found: $CODE_SOURCE"
fi
# 2. Backup database
echo "Backing up database: $DB_NAME"
mysqldump --single-transaction --quick --lock-tables=false \
  -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" | gzip > "$DB_BACKUP_FILE"
if [ $? -eq 0 ]; then
  echo " Database backup done: $(du -sh $DB_BACKUP_FILE | awk '{print $1}')"
else
  echo " Database backup failed"
  exit 1
fi
# 3. Record backup info
echo "$TIMESTAMP: code_size=$(du -sb $CODE_BACKUP_DIR 2>/dev/null | awk '{print $1}'), db_size=$(du -sb $DB_BACKUP_FILE | awk '{print $1}')" \
  >> "${BACKUP_ROOT}/backup_history.log"
# 4. Clean old backups (keep latest N)
echo "Cleaning old backups (retain latest $KEEP_COUNT)..."
# Code backups
ls -dt ${BACKUP_ROOT}/code/*/ 2>/dev/null | tail -n +$((KEEP_COUNT + 1)) | while read old_dir; do
  echo "  Deleting old code backup: $old_dir"
  rm -rf "$old_dir"
done
# Database backups
ls -dt ${BACKUP_ROOT}/db/*.sql.gz 2>/dev/null | tail -n +$((KEEP_COUNT + 1)) | while read old_file; do
  echo "  Deleting old database backup: $old_file"
  rm -f "$old_file"
done
echo "Backup completed! Timestamp: $TIMESTAMP"

Configuration Notes

CODE_SOURCE

: Code directory path BACKUP_ROOT: Backup storage root DB_NAME: Database name DB_USER / DB_PASS: Database credentials (avoid plaintext; use ~/.my.cnf)

Secure credential setup in ~/.my.cnf:

[client]
user=backup_user
password=your_password

Then simplify the dump command to mysqldump --single-transaction "$DB_NAME" | gzip > ... without password flags.

rollback.sh — Interactive Version Rollback

Usage: ./rollback.sh <project_name>. Lists all backup versions, shows whether each has a matching database backup, prompts for index, confirms, then executes.

#!/bin/bash
set -e
PROJECT_NAME="$1"
if [ -z "$PROJECT_NAME" ]; then
  echo "Error usage: $0 <project_name>"
  exit 1
fi
# ========== Config ==========
CODE_SOURCE="/data/www/${PROJECT_NAME}"
BACKUP_ROOT="/data/backups/${PROJECT_NAME}"
DB_NAME="${PROJECT_NAME}_db"
DB_USER="backup_user"
DB_PASS="your_password"
# ============================
echo "=========================================="
echo "  Project: $PROJECT_NAME"
echo "  Available backup versions"
echo "=========================================="
# List code backup directories
CODE_BACKUPS=($(ls -dt ${BACKUP_ROOT}/code/*/ 2>/dev/null | xargs -n 1 basename))
if [ ${#CODE_BACKUPS[@]} -eq 0 ]; then
  echo "Error! No backups found!"
  exit 1
fi
echo ""
echo "Available backup versions:"
for i in "${!CODE_BACKUPS[@]}"; do
  DB_FILE="${BACKUP_ROOT}/db/${CODE_BACKUPS[$i]}.sql.gz"
  if [ -f "$DB_FILE" ]; then
    DB_STATUS="yes"
  else
    DB_STATUS="error"
  fi
  echo "  [$i] ${CODE_BACKUPS[$i]}  $DB_STATUS"
done
echo ""
read -p "Select version index to rollback (0-$(( ${#CODE_BACKUPS[@]} - 1 ))): " SELECTED_INDEX
if ! [[ "$SELECTED_INDEX" =~ ^[0-9]+$ ]] || [ $SELECTED_INDEX -ge ${#CODE_BACKUPS[@]} ]; then
  echo "error Invalid selection"
  exit 1
fi
SELECTED_VERSION="${CODE_BACKUPS[$SELECTED_INDEX]}"
CODE_BACKUP_PATH="${BACKUP_ROOT}/code/${SELECTED_VERSION}"
DB_BACKUP_FILE="${BACKUP_ROOT}/db/${SELECTED_VERSION}.sql.gz"
echo ""
echo "=========================================="
echo "  Preparing rollback to version: $SELECTED_VERSION"
echo "  Code: $CODE_BACKUP_PATH"
echo "  Database: $DB_BACKUP_FILE"
echo "=========================================="
echo ""
read -p "warn Confirm rollback? (y/n): " CONFIRM
if [ "$CONFIRM" != "y" ] && [ "$CONFIRM" != "Y" ]; then
  echo "error Cancelled"
  exit 1
fi
# ========== Execute Rollback ==========
# 1. Backup current state before rollback (safety net)
echo "Backing up current state before rollback..."
CURRENT_BACKUP_TIMESTAMP=$(date +"%Y%m%d_%H%M%S_PRE_ROLLBACK")
mkdir -p "${BACKUP_ROOT}/pre_rollback"
cp -r "$CODE_SOURCE" "${BACKUP_ROOT}/pre_rollback/${CURRENT_BACKUP_TIMESTAMP}" 2>/dev/null || echo "  Code directory not found, skipping"
mysqldump --single-transaction -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" 2>/dev/null | gzip > "${BACKUP_ROOT}/pre_rollback/${CURRENT_BACKUP_TIMESTAMP}.sql.gz" || echo "  Database backup failed, skipping"
echo "Pre-rollback backup completed"
# 2. Restore code
echo "Restoring code..."
if [ -d "$CODE_BACKUP_PATH" ]; then
  rm -rf "$CODE_SOURCE"
  cp -r "$CODE_BACKUP_PATH" "$CODE_SOURCE"
  echo "Code restore completed"
else
  echo "Code backup not found: $CODE_BACKUP_PATH"
  exit 1
fi
# 3. Restore database
echo "Restoring database..."
if [ -f "$DB_BACKUP_FILE" ]; then
  gunzip -c "$DB_BACKUP_FILE" | mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME"
  echo " Database restore completed"
else
  echo "Database backup not found: $DB_BACKUP_FILE"
  exit 1
fi
echo ""
echo "Rollback completed! Version: $SELECTED_VERSION"
echo "Rollback record saved: ${BACKUP_ROOT}/rollback_history.log"
echo "$(date +"%Y%m%d_%H%M%S"): Rolled back to $SELECTED_VERSION" >> "${BACKUP_ROOT}/rollback_history.log"

Key Design Decisions

Pre-rollback snapshot : Current state is backed up first so a mistaken rollback can be reverted.

Database completeness indicator : Each listed version shows whether its matching .sql.gz exists.

Interactive selection : Operator picks an index number; no need to memorize timestamps.

list_backups.sh — List Backups with Sizes

Usage: ./list_backups.sh <project_name>. Outputs a formatted table of version, code size, database size, and status.

#!/bin/bash
PROJECT_NAME="$1"
BACKUP_ROOT="/data/backups/${PROJECT_NAME}"
echo "=========================================="
echo "  Project: $PROJECT_NAME"
echo "  Backup list"
echo "=========================================="
echo ""
printf "%-20s %-10s %-12s
" "Version" "CodeSize" "DBSize"
echo "------------------------------------------"
ls -dt ${BACKUP_ROOT}/code/*/ 2>/dev/null | while read code_dir; do
  version=$(basename "$code_dir")
  code_size=$(du -sh "$code_dir" 2>/dev/null | awk '{print $1}')
  db_file="${BACKUP_ROOT}/db/${version}.sql.gz"
  if [ -f "$db_file" ]; then
    db_size=$(du -sh "$db_file" | awk '{print $1}')
    status=""
  else
    db_size="-"
    status=""
  fi
  printf "%-20s %-10s %-12s %s
" "$version" "$code_size" "$db_size" "$status"
done

Case Study: Order Service Rollback in Under 3 Minutes

Scenario : 02:00 — Deploy order-service v2.3.0. 02:05 — Monitoring alerts: order creation error rate hits 30%.

Timeline :

02:06 — Log into bastion host, navigate to backup directory.

02:07 — Run ./list_backups.sh order-service, see versions:

Version            CodeSize  DBSize
20250615_020001    156M      89M      
20250612_100000    152M      85M      
20250610_080000    148M      82M

Select latest pre-deployment version 20250615_020001.

02:08 — Run ./rollback.sh order-service, choose index 0, confirm with y.

02:09 — Rollback completes; script restores code and database automatically.

02:10 — Verify order API healthy; notify team: "Rolled back to v2.2.0, investigating."

Total time: < 3 minutes . Business impact minimized; no frantic manual commands.

Post-mortem : Root cause was a missing index in a new SQL statement, exhausting the connection pool. Fix applied, re-deployed with EXPLAIN validation.

Common Pitfalls & Recommendations

Does mysqldump lock tables? With --single-transaction (InnoDB), no locks; business reads/writes continue. MyISAM not supported — verify engine first.

Where to store backups? Not on the system disk ( / or /var). Mount a dedicated data disk (e.g., /data). If only one disk, monitor backup directory size and alert on threshold.

Compression? Script uses gzip; typical reduction to 10–20% of original size.

Large databases (hundreds of GB)? mysqldump too slow. Use mydumper/myloader (parallel) or Percona XtraBackup (physical backup). Separate article planned.

Schedule automatic backups via cron:

0 3 * * * /usr/local/bin/backup.sh order-service 7 >> /var/log/backup.log 2>&1

Runs daily at 03:00, retains 7 versions. For daily code changes, 7 covers a week; for stable code, 3 may suffice.