How to Recover a CentOS Server When SSH Stops Working: Step‑by‑Step Fix

On a Monday, a CentOS server became unreachable via SSH while VNC still worked; web and database services also failed. Log inspection with last showed that login records before the 2nd were cleared, the sshd binary was modified on Saturday night, and the server was remotely rebooted at 2 am on Sunday.

Reviewing /var/log/messages after the reboot revealed that iptables rules were active and the system had endured a large number of SSH brute‑force scans. The test environment hosted Oracle and Squid, and temporary iptables changes had been applied; after the reboot the iptables service started again, but some system files were altered.

Solution

1. Change the root password.

2. Reinstall SSH and restrict access. Because the sshd binary had been altered, the OpenSSH packages were removed and reinstalled. The installed packages were identified with: rpm -qa | grep ssh Resulting packages: openssh-clients, openssh-server, openssh, openssh-askpass. They were removed in dependency order:

yum remove openssh-askpass openssh openssh-server openssh-clients

After removal, the packages were reinstalled using: yum install openssh-askpass openssh-server openssh-clients During installation, an “Operation not permitted” error occurred because the sshd binary had immutable and append attributes set. The hidden attributes were inspected and cleared with: chattr -ia /usr/sbin/sshd After clearing the attributes, yum install openssh-server succeeded.

3. Configure SSH login control. Set allowed internal IP addresses and define whitelist/blacklist rules.

Configure iptables for SSH

The required iptables syntax is:

iptables [-t table] [-A|I|D|R chain] [-i interface] [-p protocol] [-s source] [-d destination] [--dport port] -j action

The filter table with INPUT, OUTPUT, and FORWARD chains is used. For complex rule sets, a shell script can be written and the SSH service restarted after applying the rules.

After configuring, save the rules with /etc/rc.d/init.d/iptables save and apply them using service iptables restart.