Essential Linux Ops Lessons: Avoid Disasters with Proven Practices

1. Online Operation Standards

Test changes before applying them to production.

Double‑check every command before hitting Enter.

Avoid simultaneous edits by multiple operators.

Always back up before modifying files; comment out original settings before changing them.

2. Data Handling

Use rm -rf with extreme caution.

Backup is more important than anything else.

Stability outweighs speed; prioritize reliable configurations.

Maintain strict confidentiality for sensitive data.

3. Security

Change the default SSH port and disable root login.

Employ regular users with key‑based authentication, sudo rules, and IP restrictions.

Activate a firewall using a default‑deny policy and open only required ports.

Apply fine‑grained permission controls; run services with the least privileged accounts.

Deploy intrusion detection and centralized log monitoring for critical files and services.

4. Daily Monitoring

Monitor system health (CPU, memory, disk, network) and OS logs.

Track service status for web, database, load balancers, etc.

Collect and analyze logs for hardware, OS, and application errors.

5. Performance Tuning

Understand the underlying runtime mechanisms of software before tweaking parameters.

Follow a structured tuning workflow: identify bottlenecks, analyze logs, define a tuning direction, then adjust parameters, starting with hardware/OS before the database.

Change only one parameter at a time to isolate effects.

Conduct benchmark tests to verify improvements, referencing resources such as "High Performance MySQL".

6. Ops Mindset

Maintain composure, especially under pressure; avoid critical operations when stressed.

Take responsibility for data; always have backups.

Investigate root causes thoroughly rather than applying quick fixes.

Distinguish between test and production environments and avoid cross‑contamination.

Overall, these reflections summarize personal experiences and best‑practice recommendations for reliable Linux operations.