Essential Ops Lessons: Prevent Data Disasters and Boost Server Reliability

1. Online Operation Standards

1.1 Test Before Use

When learning Linux on virtual machines, I became accustomed to risky shortcuts; once I changed the SSH daemon configuration without testing and locked myself out, only a backup of the config saved me.

A similar mistake with rsync caused accidental deletion of production data because the source and destination directories were swapped, highlighting the critical need for testing and backups.

1.2 Confirm Before Pressing Enter

Commands like rm -rf /var can be disastrous; a single slip can cause severe damage, so always double‑check before executing destructive commands.

1.3 Avoid Multiple Operators

When several people edit the same server simultaneously, configuration changes can conflict, making root‑cause analysis difficult; a single‑operator approach reduces confusion.

1.4 Backup Before Changes

Always back up files (e.g., configuration files) before modifying them; a proper backup would have prevented data loss in the rsync incident.

2. Data Handling

2.1 Use rm -rf Sparingly

Even a small mistake with rm -rf can erase critical databases; extreme caution is required.

2.2 Backup Is Paramount

Regular backups are essential; in my former company, third‑party payment systems were backed up every two hours, while a loan platform backed up every 20 minutes.

2.3 Prioritize Stability

Stability outweighs speed; avoid deploying untested software in production, and prefer proven stacks.

2.4 Protect Confidentiality

Data confidentiality is vital given the prevalence of leaks and backdoors.

3. Security Practices

3.1 SSH Hardening

Change the default port, disable root login, use key‑based authentication with sudo, restrict IPs, and employ host‑deny tools to block repeated attacks.

3.2 Firewall Configuration

Enable a firewall in production and follow the principle of least privilege: drop everything by default and allow only necessary ports.

3.3 Fine‑Grained Permissions

Run services with non‑root users whenever possible and limit permissions to the minimum required.

3.4 Intrusion Detection and Log Monitoring

Use third‑party tools to watch critical files (e.g., /etc/passwd, /etc/my.cnf) and centralize log monitoring for security‑related events; timely alerts aid post‑incident analysis.

4. Routine Monitoring

4.1 System Health Monitoring

Track hardware metrics such as CPU, memory, disk, and network usage, as well as OS login and file integrity checks.

4.2 Service Monitoring

Monitor web, database, and load‑balancer services to quickly detect performance bottlenecks.

4.3 Log Monitoring

Collect and analyze system, application, and hardware logs to enable proactive issue detection.

5. Performance Tuning

5.1 Understand Underlying Mechanisms

Before tweaking parameters, study how software (e.g., Nginx vs. Apache) works internally; otherwise tuning is blind.

5.2 Structured Tuning Process

Identify bottlenecks via logs, define a tuning plan, and address hardware/OS issues before adjusting database settings.

5.3 Change One Parameter at a Time

Isolate the impact of each change to avoid confusion.

5.4 Benchmark Testing

Use benchmark tests to verify that tuning improves performance and matches real‑world workloads; reference resources like "High Performance MySQL" for guidance.

6. Ops Mindset

6.1 Control Emotions

Stay calm during high‑pressure moments (e.g., accidental rm -rf) and avoid making critical changes when stressed.

6.2 Take Responsibility for Data

Production data must be treated with utmost care; lack of backups leads to severe consequences.

6.3 Investigate Root Causes

After fixing an issue, continue probing to uncover underlying problems, such as memory shortages causing MySQL OOM kills.

6.4 Separate Test and Production Environments

Always verify operations on the appropriate environment and limit simultaneous sessions.