Essential Linux Ops Practices to Prevent Disasters

Online Operational Standards

After many incidents like data loss, website hijacking, accidental deletion of database files, and hacker attacks, the author shares a concise set of Linux operating habits aimed at keeping production environments safe.

Test‑First Mentality

When learning Linux on virtual machines, it’s easy to develop reckless habits. The author recounts trying to switch to Xshell with key‑based SSH login without testing, which locked him out of the server until a backup of sshd_config was restored.

File Synchronization Pitfalls

Using rsync can be faster than rm -rf, but a mistaken source‑destination order can delete valuable data. The lack of testing and backup led to irreversible loss in production.

Key Safety Rules

Confirm before pressing Enter : Mistyping commands like rm -rf /var can instantly cripple a system.

Avoid multi‑person operations : When several people edit the same server simultaneously, configuration drift and confusion are common.

Backup before changes : Always copy configuration files (e.g., .conf) and comment out original options before modifying them.

Data‑Related Practices

Use rm‑rf sparingly : Accidental deletions of critical directories can cause massive loss.

Prioritize backups : Regularly back up databases; the author cites a payment platform that backs up every two hours and a loan platform every 20 minutes.

Stability over speed : Prefer proven, stable software stacks; test new components (e.g., Nginx + PHP‑FPM) in non‑production environments first.

Maintain confidentiality : Protect sensitive data against leaks and backdoors.

Security Measures

SSH hardening : Change the default port, disable root login, enforce key‑based authentication, use sudo, restrict IPs, and employ host‑based denial tools.

Firewall : Apply a default‑deny policy and only open required service ports.

Fine‑grained permissions : Run services with the least privileged accounts, never as root.

Intrusion detection & log monitoring : Deploy third‑party tools to watch critical files (e.g., /etc/passwd, /etc/my.cnf) and centralize logs for alerts on suspicious activity.

Daily Monitoring

System health : Track hardware utilization (CPU, memory, disk, network) and OS login activity.

Service metrics : Monitor web, database, and load‑balancer (LVS) indicators to spot performance bottlenecks.

Log oversight : Combine security‑related logs with application and hardware alerts to react quickly when issues arise.

Performance Tuning

Understand underlying mechanisms : Know why Nginx is faster than Apache before tweaking parameters.

Follow a tuning framework : Identify bottlenecks, analyze logs, set a clear direction, and adjust OS/hardware before touching database settings.

Change one parameter at a time : Isolate the impact of each modification.

Benchmark testing : Use realistic workloads to verify that tuning improves performance without harming stability.

Ops Mindset

Control emotions : Avoid making critical changes when stressed; keep a calm recovery plan.

Take responsibility for data : Unbacked data loss is severe; always assume accountability.

Root‑cause analysis : After fixing an issue, investigate why it happened (e.g., OOM kills due to insufficient memory).

Separate test and production : Never run risky commands on live systems without proper validation.