50 Essential Ops Troubleshooting & Fix Techniques for Rapid Issue Resolution

1. System Layer

Check system logs :

Tip : Review journalctl and files under /var/log to find clues.

Fix : Adjust service configuration based on log findings and restart the service.

High load investigation :

Tip : Use top or htop to analyze CPU, memory, and I/O usage.

Fix : Optimize load processes, adjust priorities, or add resources.

Memory leak detection :

Tip : Use free , vmstat to view memory usage, and valgrind to analyze process memory.

Fix : Restart the process and resolve the leak.

Disk space shortage :

Tip : Run df -h to check usage and du -sh to locate large files.

Fix : Delete unnecessary files, clean logs, or expand disk capacity.

Service fails to start :

Tip : Use systemctl to view service status and related logs.

Fix : Check dependencies and configuration errors, then restart.

Kernel parameter tuning :

Tip : Use sysctl to view and modify kernel parameters.

Fix : Optimize TCP buffers, max connections, etc., to improve performance.

Process crashes :

Tip : Examine kernel logs with dmesg to find crash reasons.

Fix : Investigate resource exhaustion or code bugs, then restart.

CPU bottleneck analysis :

Tip : Use mpstat or sar to check CPU usage.

Fix : Optimize application code, adjust load balancing, or add CPU cores.

Filesystem issues :

Tip : Run fsck to detect filesystem errors.

Fix : Execute fsck during reboot to repair.

Excessive swap usage :

Tip : Check swap with vmstat .

Fix : Add physical memory and adjust swap policies.

2. Network Layer

Network connectivity check :

Tip : Use ping and traceroute to verify routes.

Fix : Correct network configuration and firewall rules.

Port conflict :

Tip : Inspect ports with netstat or ss .

Fix : Terminate the occupying process or change the application port.

Firewall issues :

Tip : Review rules using iptables or firewalld .

Fix : Modify rules to open necessary ports.

DNS resolution problems :

Tip : Query with nslookup or dig .

Fix : Check local DNS settings or switch DNS servers.

Network congestion :

Tip : Analyze traffic using iftop or nload .

Fix : Limit heavy traffic tasks, optimize topology, or upgrade bandwidth.

TCP connection timeout :

Tip : Check connections with netstat or ss .

Fix : Adjust TCP timeout parameters and connection pool settings.

High bandwidth usage :

Tip : View usage via iftop .

Fix : Restrict bandwidth‑heavy processes or users and rebalance allocation.

ARP conflicts :

Tip : Detect with arp -a .

Fix : Correct IP address assignments to avoid clashes.

MTU mismatch :

Tip : Test using ping -M do -s .

Fix : Adjust MTU settings to match device parameters.

SSL certificate issues :

Tip : Inspect with openssl tools.

Fix : Update or regenerate the certificate.

3. Application Layer

Application service crash :

Tip : Review log files for pre‑crash entries.

Fix : Optimize configuration or fix code errors to ensure stability.

High concurrency bottlenecks :

Tip : Check concurrent connections with netstat or sar .

Fix : Add load‑balancing nodes and tune application code and DB queries.

Application deadlock :

Tip : Debug with strace or gdb .

Fix : Correct logic to avoid concurrent lock situations.

Slow application startup :

Tip : Trace system calls using strace .

Fix : Streamline the startup sequence to reduce load time.

Oversized application logs :

Tip : Periodically check size and rotate with logrotate .

Fix : Adjust log level and clean up old logs.

Application port conflict :

Tip : Identify with lsof or netstat .

Fix : Release the occupied port or change the app’s configuration.

Connection‑pool exhaustion :

Tip : Look for pool‑exhaustion errors in app logs.

Fix : Increase pool size or optimize queries.

Misconfiguration :

Tip : Verify parameters in config files.

Fix : Correct the file and reload the service.

Application timeout issues :

Tip : Test response time with curl or ab .

Fix : Increase timeout settings and speed up DB queries.

Dependent service unavailable :

Tip : Probe with curl or telnet .

Fix : Check the dependent service’s status and restart if needed.

4. Database Layer

Database connection failure :

Tip : Verify port, user permissions, and network reachability.

Fix : Correct permission or network settings.

Slow queries :

Tip : Run EXPLAIN to view execution plans.

Fix : Optimize SQL, add indexes, or partition tables.

Database deadlocks :

Tip : Use engine‑specific lock status commands, e.g., SHOW ENGINE INNODB STATUS .

Fix : Refine transaction handling to avoid long locks.

Performance bottlenecks :

Tip : Employ mysqltuner or built‑in monitoring tools.

Fix : Increase cache, tune queries, or upgrade hardware.

Master‑slave replication lag :

Tip : Inspect replication status on both ends.

Fix : Lighten master load, add slaves, or adjust replication strategy.

Table locking :

Tip : Query SHOW PROCESSLIST for lock info.

Fix : Optimize queries and avoid large batch operations.

Backup failures :

Tip : Review backup logs to pinpoint cause.

Fix : Expand storage or modify backup strategy.

Database I/O problems :

Tip : Use iostat to monitor I/O usage.

Fix : Deploy SSDs or add RAID arrays to boost I/O performance.

Insufficient tablespace :

Tip : Run SHOW TABLE STATUS to see usage.

Fix : Expand tablespace and purge unused data.

Too many connections :

Tip : Check with SHOW STATUS .

Fix : Raise max connections or improve pool management.

5. Security & Permission Management

Permission errors blocking access :

Tip : Use chmod and chown to adjust file permissions.

Fix : Set appropriate permissions for users.

SSH login failures :

Tip : Examine /var/log/auth.log or journalctl .

Fix : Review SSH config and firewall rules.

Brute‑force protection :

Tip : Deploy fail2ban to monitor suspicious attempts.

Fix : Configure auto‑ban policies.

Overly strict firewall rules :

Tip : Review with iptables or firewalld .

Fix : Open required ports and balance policies.

Regular password changes :

Tip : Enforce periodic password policies.

Fix : Require users to update passwords regularly.

Log auditing :

Tip : Use auditd to capture user actions.

Fix : Review logs routinely for anomalies.

File integrity checks :

Tip : Run tripwire or aide .

Fix : Respond promptly to any integrity alerts.

Application vulnerability scanning :

Tip : Scan with OpenVAS or Nessus .

Fix : Patch identified vulnerabilities.

ACL management :

Tip : View and modify ACLs using setfacl .

Fix : Apply sensible access controls to prevent privilege abuse.

Log rotation failures :

Tip : Verify logrotate configuration.

Fix : Adjust rotation policies to ensure regular archiving.

Conclusion

These 50 operations troubleshooting and remediation techniques span system, network, application, database, and security domains; mastering them enables engineers to quickly pinpoint issues, apply effective fixes, and keep infrastructure stable and secure.