10 Essential Shell Scripts for Efficient Log Management

In operations work, log handling is a critical task for monitoring system status, troubleshooting failures, and performance tuning. The following ten Bash scripts illustrate common log‑processing techniques; they should be adapted to the specific environment before use.

1. Real‑time Log Monitoring and Alert (monitor_log_and_alert.sh)

Function: Continuously watches a log file and sends an alert email when a specified error keyword appears.

#!/bin/bash

LOG_FILE="/path/to/your/logfile.log"
KEYWORD="ERROR"
EMAIL="[email protected]"

tail -F $LOG_FILE |
while read line; do
  if echo "$line" | grep -q "$KEYWORD"; then
    echo "Alert: $line" | mail -s "Error Detected in $LOG_FILE" $EMAIL
  fi
done

2. Log Rotation by Date (log_rotate.sh)

Function: Simulates the logrotate utility by renaming the current log with the current date and creating a new empty file.

#!/bin/bash

LOG_FILE="/path/to/your/logfile.log"
DATE=$(date +%Y%m%d)

# Check if yesterday's log exists
if [ ! -f "$LOG_FILE.$DATE" ]; then
  mv "$LOG_FILE" "$LOG_FILE.$DATE"
  touch "$LOG_FILE"
  echo "Rotated log file to $LOG_FILE.$DATE"
fi

3. Aggregate Errors Across Servers (aggregate_errors.sh)

Function: Collects error logs from multiple servers and aggregates them for further analysis.

#!/bin/bash

SERVERS=("server1" "server2" "server3")
ERROR_DIR="/path/to/error_logs"
mkdir -p "$ERROR_DIR"

for SERVER in "${SERVERS[@]}"; do
  SSH_CMD="ssh $SERVER 'cat /path/to/logs/error.log'"
  ERRORS=$(eval $SSH_CMD)
  echo "$ERRORS" | grep 'ERROR' >> "$ERROR_DIR/${SERVER}_errors.log"
done
# Optional further analysis of these error logs
# ...

4. Timestamp Conversion (timestamp_convert.sh)

Function: Converts UNIX timestamps in a log file to a human‑readable date format.

#!/bin/bash

LOG_FILE="/path/to/your/logfile_with_timestamps.log"
OUTPUT_FILE="/path/to/converted_log.log"

while IFS= read -r line; do
  TIMESTAMP=$(echo $line | cut -d' ' -f1)
  MESSAGE=$(echo $line | cut -d' ' -f2-)
  HUMAN_READABLE_DATE=$(date -d @$TIMESTAMP "+%Y-%m-%d %H:%M:%S")
  echo "$HUMAN_READABLE_DATE $MESSAGE" >> "$OUTPUT_FILE"
done < "$LOG_FILE"

5. IP Address Statistics (ip_address_statistics.sh)

Function: Counts occurrences of each IP address in an access log and outputs a sorted list.

#!/bin/bash

LOG_FILE="/path/to/your/access.log"
# Extract the first field (IP), sort, count, then sort numerically descending
awk '{print $1}' $LOG_FILE | sort | uniq -c | sort -nr

6. Request Type Analysis (request_type_analysis.sh)

Function: Analyzes the proportion of different HTTP request methods (e.g., GET, POST) in a log file.

#!/bin/bash

LOG_FILE="/path/to/your/access.log"
declare -A request_types

while IFS= read -r line; do
  method=$(echo $line | awk '{print $6}') # assumes method is the 6th field
  if [[ "${request_types[$method]}" ]]; then
    ((request_types[$method]++))
  else
    request_types[$method]=1
  fi
done < "$LOG_FILE"

echo "Request Type Statistics:"
for method in "${!request_types[@]}"; do
  echo "$method: ${request_types[$method]}"
done

7. Response Time Analysis (response_time_analysis.sh)

Function: Buckets request response times into ranges and counts how many requests fall into each bucket.

#!/bin/bash

LOG_FILE="/path/to/your/access.log"
declare -A response_times
bins=("0-100ms" "101-200ms" "201-500ms" "501-1000ms" "1000ms+")

while IFS= read -r line; do
  response_time=$(echo $line | awk '{print $10}') # assumes response time is the 10th field
  if (( $(echo "$response_time <= 100" | bc -l) )); then
    bin="0-100ms"
  elif (( $(echo "$response_time <= 200" | bc -l) )); then
    bin="101-200ms"
  elif (( $(echo "$response_time <= 500" | bc -l) )); then
    bin="201-500ms"
  elif (( $(echo "$response_time <= 1000" | bc -l) )); then
    bin="501-1000ms"
  else
    bin="1000ms+"
  fi
  ((response_times[$bin]++))
 done < "$LOG_FILE"

echo "Response Time Statistics:"
for bin in "${!response_times[@]}"; do
  echo "$bin: ${response_times[$bin]}"
done

8. Log File Difference Analysis (log_diff_analysis.sh)

Function: Compares two log files and shows their differences side‑by‑side, ignoring blank lines.

#!/bin/bash

LOG_FILE1="/path/to/log1.log"
LOG_FILE2="/path/to/log2.log"
# Use diff with -B to ignore blank lines and --side-by-side for parallel view
diff -B --side-by-side $LOG_FILE1 $LOG_FILE2

9. User Activity Tracker (user_activity_tracker.sh)

Function: Extracts all log entries for a specific user and sorts them chronologically.

#!/bin/bash

LOG_FILE="/path/to/user_activity.log"
USER_NAME="target_user"
# Find lines containing the username and sort them
awk -v user="$USER_NAME" '$0 ~ user {print}' $LOG_FILE | sort

10. Log Compression and Backup (log_compress_and_backup.sh)

Function: Compresses a log directory into a tar.gz archive and optionally copies it to a remote server.

#!/bin/bash

LOG_DIR="/path/to/logs"
BACKUP_DIR="/path/to/backup"
DATE=$(date +%Y%m%d)
BACKUP_FILE="logs_backup_${DATE}.tar.gz"

# Compress the log directory
tar -czvf $BACKUP_DIR/$BACKUP_FILE $LOG_DIR

# Optional: copy the backup to a remote server using scp
# scp $BACKUP_DIR/$BACKUP_FILE username@remotehost:/path/to/remote/backup/

echo "Backup completed: $BACKUP_DIR/$BACKUP_FILE"

Each script is a minimal example; adjust paths, keywords, field positions, and remote details to match the actual log format and infrastructure.