Mastering Shell Scripting: 18 Advanced Tricks to Supercharge Ops Efficiency

Shell Scripting "18 Skills": Advanced Techniques to Boost Ops Efficiency

Introduction: From 3‑second optimization to 0.3‑second

Last year before Double‑Eleven we needed to process nearly 10 TB of logs daily. A script that originally took three hours was reduced to 20 minutes by applying parallel processing and pipeline optimization, demonstrating that Shell is more than a glue language—it is a Swiss‑army knife for ops engineers.

Below we share the pitfalls, tips, and "black‑magic" tricks we have gathered.

1. Why Shell scripts remain the ops mainstay?

In the era of containers and cloud‑native, you may wonder whether learning Shell is still worthwhile.

Answer: Absolutely necessary!

Imagine these scenarios:

3 am production alert requiring rapid diagnosis

Kubernetes pod startup failures needing bulk checks across hundreds of nodes

CI/CD pipelines with custom deployment logic

Database backup scripts that must intelligently choose strategies

In such cases Shell scripts act like a stethoscope—simple, direct, and efficient, and they run on any Linux system without extra deployment.

2. Shell scripting "18 skills" practical highlights

Skill 1: The art of parallel processing

Real case: checking disk usage on 1 000 servers.

#!/bin/bash
# Traditional serial execution, ~500 seconds
for host in $(cat servers.txt); do
    ssh $host "df -h" >> result.txt
done

# Advanced parallel processing, ~10 seconds
check_disk() {
    local host=$1
    ssh -o ConnectTimeout=5 $host "df -h" 2>/dev/null || echo "$host: connection failed"
}
export -f check_disk
# Use GNU parallel or xargs for parallelism
cat servers.txt | xargs -P 50 -I {} bash -c 'check_disk {}'

# More elegant with a process pool
MAX_JOBS=50
job_count=0
while IFS= read -r host; do
    check_disk "$host" &
    ((job_count++))
    if [ $job_count -ge $MAX_JOBS ]; then
        wait -n
        ((job_count--))
    fi
done < servers.txt
wait

Key point: Parallelism is not unlimited; set concurrency based on network bandwidth and target load.

Skill 2: Defensive programming for error handling

Production scripts must be "bullet‑proof".

#!/bin/bash
# Strict mode
set -euo pipefail
IFS=$'
\t'

# Custom error handler
error_exit() {
    echo "Error: $1" >&2
    curl -X POST https://alert.company.com/webhook -d '{"message":"Script failed: $1"}' >/dev/null 2>&1
    exit 1
}
trap 'error_exit "Error at line $LINENO"' ERR

# Smart retry
retry_command() {
    local max_attempts=${1:-3}
    local delay=${2:-1}
    local command="${@:3}"
    local attempt=1
    while [ $attempt -le $max_attempts ]; do
        if eval "$command"; then
            return 0
        fi
        echo "Command failed, retrying in $delay seconds... (attempt $attempt/$max_attempts)"
        sleep $delay
        ((attempt++))
        delay=$((delay*2))
    done
    return 1
}
# Example usage
retry_command 3 2 "curl -f https://api.example.com/health" || error_exit "API health check failed"

Skill 3: Performance optimization secrets

Shell performance checklist:

# 1. Avoid useless cat
# Bad
cat file.txt | grep "pattern"
# Good
grep "pattern" file.txt

# 2. Prefer built‑ins over external commands
# Bad
result=$(echo "$string" | sed 's/old/new/')
# Good
result=${string//old/new}

# 3. Batch processing instead of line‑by‑line loops
# Bad
while read line; do
    echo "$line" | awk '{print $2}'
 done < bigfile.txt
# Good
awk '{print $2}' bigfile.txt

# 4. Process substitution to avoid temporary files
diff <(sort file1.txt) <(sort file2.txt)

# 5. Pre‑compile regex for repeated use
regex='^[0-9]{1,3}(\.[0-9]{1,3}){3}$'
grep -E "$regex" access.log | while read ip; do
    # handle IP
 done

Skill 4: Stream‑based log analysis

Streaming large logs reduces memory consumption.

#!/bin/bash
analyze_logs() {
    tail -f /var/log/nginx/access.log | awk '
        {
            ip_count[$1]++
            if (NR % 10000 == 0) {
                print "=== Stats at " strftime("%Y-%m-%d %H:%M:%S") " ==="
                n = asorti(ip_count, sorted_ips, "@val_num_desc")
                for (i = 1; i <= 10 && i <= n; i++) {
                    print sorted_ips[i], ip_count[sorted_ips[i]]
                }
                print ""
            }
        }
    '
}
analyze_logs | while read line; do
    if echo "$line" | grep -q "^[0-9]" && [ $(echo "$line" | awk '{print $2}') -gt 1000 ]; then
        echo "High‑frequency access detected: $line"
    fi
done

Skill 5: Shell in container environments

Shell scripts are essential for Kubernetes operations.

#!/bin/bash
# Restart failing pods
kubectl get pods --all-namespaces | grep -E "CrashLoopBackOff|Error|Evicted" | awk '{print $1, $2}' |
while read namespace pod; do
    echo "Restarting pod: $namespace/$pod"
    kubectl delete pod $pod -n $namespace --grace-period=0 --force
done

# Auto‑scale deployment based on CPU
auto_scale() {
    local deployment=$1
    local namespace=${2:-default}
    local cpu_threshold=80
    cpu_usage=$(kubectl top pods -n $namespace | grep $deployment | awk '{sum+=$2} END {print sum/NR}' | sed 's/%//')
    current_replicas=$(kubectl get deployment $deployment -n $namespace -o jsonpath='{.spec.replicas}')
    if (( $(echo "$cpu_usage > $cpu_threshold" | bc -l) )); then
        new_replicas=$((current_replicas+2))
        kubectl scale deployment $deployment -n $namespace --replicas=$new_replicas
        echo "Scaled up $deployment from $current_replicas to $new_replicas"
    elif (( $(echo "$cpu_usage < 30" | bc -l) )) && [ $current_replicas -gt 2 ]; then
        new_replicas=$((current_replicas-1))
        kubectl scale deployment $deployment -n $namespace --replicas=$new_replicas
        echo "Scaled down $deployment from $current_replicas to $new_replicas"
    fi
}
# Collect pod logs with error filtering
collect_pod_logs() {
    local label_selector=$1
    local since=${2:-1h}
    kubectl get pods -l "$label_selector" -o name |
    parallel -j 10 "kubectl logs {} --since=$since 2>/dev/null | grep -E 'ERROR|FATAL|Exception' | jq -R -s 'split(\"
\") | map(select(length>0))'"
}

3. Ops "pitfalls" and lessons learned

1. Variable scope traps

# Wrong: variable changes inside a pipeline are lost
count=0
cat file.txt | while read line; do
    ((count++))
 done
echo "Lines: $count"   # always 0

# Correct
count=0
while read line; do
    ((count++))
 done < file.txt
echo "Lines: $count"

2. Handling spaces in filenames

# Dangerous
for file in $(ls *.txt); do
    rm $file   # fails on spaces
 done

# Safe
for file in *.txt; do
    [ -e "$file" ] || continue
    rm "$file"
 done

# Even safer with find
find . -maxdepth 1 -name "*.txt" -type f -print0 | xargs -0 rm

3. Password and sensitive data handling

# Never pass passwords in clear text
# Bad
mysql -u root -p123456 -e "show databases"

# Recommended: use environment variable
export MYSQL_PWD="123456"
mysql -u root -e "show databases"

# Or config file with restricted permissions
cat > ~/.my.cnf <<EOF
[client]
password=123456
EOF
chmod 600 ~/.my.cnf
mysql -u root -e "show databases"

# Or secret manager
password=$(vault kv get -field=password secret/mysql)
MYSQL_PWD="$password" mysql -u root -e "show databases"

4. The future of Shell: integration with modern tools

1. AI‑augmented ops

#!/bin/bash
diagnose_issue() {
    local error_log=$1
    local context=$(tail -n 100 "$error_log" | head -n 50)
    response=$(curl -s -X POST https://api.openai.com/v1/chat/completions \
        -H "Authorization: Bearer $OPENAI_API_KEY" \
        -H "Content-Type: application/json" \
        -d '{
            "model": "gpt-4",
            "messages": [
                {"role":"system","content":"You are an ops expert, analyze error logs and suggest solutions"},
                {"role":"user","content":"'