Essential Linux Ops Practices: Prevent Disasters and Boost Stability

Introduction

The author shares personal incidents from three‑plus years of Linux operations—data loss, server hijacking, accidental deletions—and distills lessons into actionable best‑practice guidelines.

1. Online Operation Standards

Test before using: Perform all experiments on virtual machines first; avoid transferring habits like frequent snapshot restores to production.

Confirm before pressing Enter: Mistyped commands such as rm -rf /var or reversed rsync paths can erase critical data instantly.

Avoid multi‑person edits: Simultaneous changes by several operators lead to configuration drift and confusion.

Backup before any change: Always copy configuration files (e.g., .conf) and keep database snapshots; a missing backup turns a simple mistake into a disaster.

2. Data Handling

Use rm -rf with extreme caution: A single slip can delete entire databases; verify intent rigorously.

Prioritize backups: The author’s companies performed full backups every two hours (payment gateway) or every 20 minutes (loan platform).

Stability over speed: Prefer proven, stable software stacks in production; avoid untested versions of Nginx, PHP‑FPM, etc.

Confidentiality: Protect sensitive data and prevent leaks through proper access controls and network hardening.

3. Security Measures

Change the default SSH port (recognizing that determined attackers can still scan).

Disable direct root login.

Adopt a normal‑user + key‑based authentication + sudo rules + IP restrictions model.

Deploy intrusion‑prevention tools (e.g., hostdeny) to block repeated brute‑force attempts.

Audit /etc/passwd and other critical accounts.

Enable a firewall with a default‑deny policy, opening only required ports.

Apply least‑privilege principles: run services as non‑root users and limit permissions.

Use third‑party intrusion detection and log monitoring to watch critical files ( /etc/passwd, /etc/my.cnf, etc.) and log streams ( /var/log/secure, /etc/log/message).

4. Daily Monitoring

System health: Track CPU, memory, disk, network, and OS login activity.

Service health: Monitor web, database, and load‑balancer metrics to spot performance bottlenecks early.

Log monitoring: Collect hardware, OS, and application error logs; alerts become crucial when issues arise.

5. Performance Tuning

Understand the underlying execution mechanisms of software (e.g., why Nginx outperforms Apache) before tweaking parameters.

Follow a tuning framework: diagnose the bottleneck, analyze logs, define a direction, then adjust OS/hardware before touching database configs.

Change only one parameter at a time to isolate effects.

Conduct benchmark tests to verify improvements and ensure they reflect real‑world workloads.

6. Ops Mindset

Control emotions: Avoid making critical changes when stressed; pause if you feel rushed.

Take responsibility for data: Recognize that production data is not a playground; lack of backup has severe consequences.

Root‑cause analysis: After fixing an issue, investigate why it happened (e.g., OOM kills, MySQL bugs).

Separate test and production environments: Verify operations on test machines and limit open terminals during critical tasks.