Essential Ops Practices: Prevent Disasters with Backups, Security, and Tuning

1. Online Operation Standards

When learning Linux on virtual machines, the author became accustomed to risky shortcuts that later caused serious problems on real servers. Always test changes in a safe environment, confirm actions before pressing Enter, and avoid multiple people editing the same configuration simultaneously.

Backup configuration files (e.g., .conf) before modifying them.

Comment out original options and add new ones instead of overwriting.

Regularly back up databases to mitigate accidental rsync deletions.

2. Data Handling

Commands like rm -rf / can erase critical data instantly; use them only with extreme caution. Backups are paramount—without them, a single mistake can cause irreversible loss.

Stability should be prioritized over raw speed; a reliable system is more valuable than a marginal performance gain.

Confidentiality is also essential, especially when handling sensitive data such as payment or loan information.

3. Security Measures

SSH hardening :

Change the default port.

Disable root login.

Use regular users with key authentication, sudo rules, IP restrictions, and user limits.

Deploy tools like HostDeny to block brute‑force attempts.

Audit /etc/passwd for unauthorized users.

Firewall : Enable it in production and follow the principle of least privilege—drop all traffic by default and allow only required ports.

Fine‑grained permissions : Run services with the lowest possible privileges, never as root when a normal user suffices.

Intrusion detection and log monitoring :

Use third‑party tools to watch critical files (e.g., /etc/passwd, /etc/my.cnf, /etc/httpd/conf).

Centralize logs and monitor /var/log/secure, /etc/log/message, FTP activity, etc.

Block repeated port scans by adding offenders to host.deny.

4. Daily Monitoring

System resource monitoring (CPU, memory, disk, network) and login/file changes help predict hardware failures and guide tuning.

Service monitoring (Web, DB, LVS, etc.) tracks key metrics to quickly spot performance bottlenecks.

Log monitoring captures OS and application errors; without it, troubleshooting becomes reactive.

5. Performance Tuning

Understand the runtime mechanisms of software (e.g., why Nginx is fast) before adjusting parameters; reading source code may be necessary.

Adopt a tuning framework: diagnose the bottleneck, analyze logs, define a direction, then adjust, starting with hardware/OS before touching database settings.

Change only one parameter at a time to isolate effects.

Conduct benchmark tests that reflect real‑world workloads; reference materials such as the third edition of “High Performance MySQL”.

6. Ops Mindset

Maintain composure under pressure; avoid critical operations when stressed.

Take responsibility for data—unbacked data loss can be catastrophic.

Investigate root causes thoroughly; repeated fixes without understanding the underlying issue waste time.

Distinguish test and production environments, limit open windows, and verify the target machine before executing risky commands.