Essential Ops Checklist: Prevent Data Loss, Secure Servers, and Optimize Performance

1. Safe Operational Practices

Never execute commands on production without prior testing; the author recounts a failed SSH configuration change that locked them out, emphasizing the need for backups of critical files like sshd_config. Always double‑check commands before pressing Enter, especially destructive ones such as rm -rf, and avoid simultaneous edits by multiple operators to prevent configuration drift.

2. Data Handling and Backup

Misusing rsync can delete source data if the direction is reversed, highlighting the importance of backups before any sync operation. Regular backups are crucial: the author cites a payment platform that backs up every two hours and a loan platform that backs up every 20 minutes. Backup is presented as the single most important safeguard.

3. Security Hardening

Change the default SSH port (recognizing that determined attackers can still scan).

Disable direct root login.

Adopt key‑based authentication with sudo rules, IP restrictions, and user limits.

Deploy host‑based intrusion‑prevention tools that block repeated failed attempts.

Maintain a minimal‑allow firewall, only opening required service ports, and enforce fine‑grained permissions so services run as non‑root users whenever possible.

4. Continuous Monitoring

Implement system‑level monitoring (CPU, memory, disk, network, login activity) and service‑level monitoring (web, database, load balancer) to detect performance bottlenecks early. Log monitoring should capture hardware, OS, and application errors, as well as security‑relevant events such as changes to /etc/passwd or configuration files.

5. Performance Tuning Framework

Understand the underlying runtime mechanisms of software (e.g., why Nginx outperforms Apache) before tweaking parameters. Follow a structured tuning process: identify the bottleneck, analyze logs, define a tuning direction, and adjust one parameter at a time. Conduct benchmark tests to verify the impact and ensure changes align with business requirements.

6. Operational Mindset

Maintain composure under pressure; avoid making critical changes when frustrated. Take responsibility for data integrity, perform thorough testing before production changes, and investigate root causes of recurring issues rather than applying superficial fixes. When data loss occurs, prioritize preserving in‑memory state, clone disks, and consider professional recovery only as a last resort.