Top 10 Linux Ops Troubleshooting Tips Every Sysadmin Should Know

As a Linux operations engineer you often encounter various issues; this article compiles ten common problems and practical solutions.

1. Shell script not executing

Problem: A script reports "bad interpreter: No such file or directory".

Cause: The script was edited on Windows, leaving CRLF line endings (\r) and ^M characters.

Solution:

Rewrite the script directly on Linux.

In vi run :%s/\r//g and :%s/^M//g (enter ^M with Ctrl+v Ctrl+m).

Use sh -x scriptname for step‑by‑step debugging.

2. Crontab output filling /var/spool/clientmqueue

Problem: The directory exceeds 100 GB because cron jobs send output via email, but sendmail is not running.

Solution:

Manually delete the files (e.g., ls | xargs rm -f).

Append >/dev/null 2>&1 to cron commands to discard output.

3. Telnet/SSH very slow

Problem: Connections to a remote host are sluggish, while ping works.

Cause: The client performs a reverse DNS lookup that fails.

Solution:

Add the hostname‑IP mapping to /etc/hosts.

Comment out or replace the non‑functional nameserver entry in /etc/resolv.conf with a working DNS server.

4. Read‑only file system error in MySQL

Problem: Creating a table fails with "ERROR 1005 … Read‑only file system".

Possible causes:

File‑system corruption.

Bad disk sectors.

Incorrect fstab configuration (wrong filesystem type or typo).

Solution:

Reboot the test machine (often restores normal state).

Remount the filesystem if appropriate (e.g., using mount -o remount,rw /path).

5. Deleted files do not free disk space

Problem: df -h shows 90 GB used, but du -sh /* totals only 30 GB.

Cause: A process still holds an open file descriptor to a deleted file.

Solution:

Restart the system or the affected service.

Identify the holding process with lsof | grep deleted and terminate it.

Release the space directly: echo > /proc/<pid>/fd/<fd>.

Truncate a still‑writing file with cat /dev/null > filename.

6. Improving find performance for cleaning temporary pictures

Problem: A nightly find /tmp -name "picture_*" -mtime +1 -exec rm -f {} \; script overloads the server.

Cause: Scanning a directory with a huge number of files is resource‑intensive.

Optimized solution:

#!/bin/sh
cd /tmp
time=$(date -d "2 day ago" "+%b %d")
ls -l | grep "picture" | grep "$time" | awk '{print $NF}' | xargs rm -rf

7. Unable to obtain gateway MAC address

Problem: ARP shows an incomplete entry for the gateway, causing network issues.

Solution: Add a static ARP entry:

arp -i bond0 -s 192.168.3.254 00:00:5e:00:01:64

8. HTTP service fails to start (port conflict)

Problem: Starting httpd reports "Address already in use" for port 7080.

Cause: The port is defined twice in configuration files ( /etc/httpd/conf/http.conf and /etc/httpd/conf.d/t.10086.cn.conf).

Solution: Comment out the duplicate Listen 7080 line in /etc/httpd/conf.d/t.10086.cn.conf and restart the service.

9. "Too many open files" error

Solution: Increase limits in /etc/security/limits.conf and /root/.bash_profile:

echo "* soft nproc 65535" >> /etc/security/limits.conf
echo "* hard nproc 65535" >> /etc/security/limits.conf
echo "* soft nofile 65535" >> /etc/security/limits.conf
echo "* hard nofile 65535" >> /etc/security/limits.conf

echo "ulimit -n 65535" >> /root/.bash_profile
echo "ulimit -u 65535" >> /root/.bash_profile

Then reboot or run ulimit -u 65535 && ulimit -n 65535.

10. ibdata1 and mysql‑bin consuming disk space

Problem: ibdata1 exceeds 120 GB and mysql‑bin logs exceed 80 GB. Background: InnoDB can use a shared tablespace (ibdata1) or independent tablespaces (one file per table). Shared tablespaces grow continuously and cannot shrink automatically; independent tablespaces allow space reclamation. Solutions:

ibdata1 too large: Dump all databases, delete the current data directory, and restore from the dump.

mysql‑bin logs too large:

Manually purge logs: PURGE MASTER LOGS TO 'mysql-bin.010'; or PURGE MASTER LOGS BEFORE '2020-12-22 13:00:00'; Configure automatic expiration in /etc/my.cnf with expire_logs_days = 30.

Applying these fixes helps keep Linux servers stable and prevents resource exhaustion.