Recovering a Linux Server Stuck in Rescue Mode and Re-mounting a Data Disk without Formatting

Earlier I encountered a server issue where an illegal reboot caused the machine to enter rescue mode and the data disk failed to mount. I share the whole troubleshooting process to help others avoid similar problems.

1. Problem Background

A colleague reported that a server could not be connected and was stuck on the Xshell connection screen. Pressing Ctrl+Alt+] switched to the local shell, confirming the connection failure.

Using the server's remote management (iKVM HTML5), I accessed the console and performed a manual reboot, after which Xshell could reconnect. However, simple commands returned I/O errors, and soon the server stopped responding entirely, entering rescue mode.

2. Rescue Mode Commands

In rescue mode the following commands are useful: journalctl -xb – view system logs. systemctl reboot – reboot the system. systemctl default or ^D – attempt to return to the default target.

Enter the root password to gain a shell.

The logs indicated that mounting the filesystem was the root cause.

3. Initial Diagnosis with df -h

Running df -h showed that the expected /dev/sdb1 partition was not mounted on /bigdata. Attempts to unmount and remount produced various errors.

4. Disk Status Check

RAID card status displayed as Online, and the server showed an initialization screen after a reboot, confirming the hardware was functional.

5. Repairing the XFS Filesystem

Since the partition uses XFS, the appropriate repair command is: xfs_repair -L /dev/sdb1 The repair can take a long time depending on data usage, so it was run in the background and monitored until completion.

6. Re‑mounting the Partition

After repair, the partition was mounted with: mount /dev/sdb1 /bigdata Verification with df -h confirmed the mount succeeded.

7. Mounting a Disk without Existing Data

If the disk contains no valuable data, the following steps can be used:

ll /dev/disk/by-path   # view disk name
fdisk -l               # view disk info
lsblk                  # list block devices

parted /dev/sdb mklabel gpt
parted /dev/sdb mkpart primary 0 100%
mkfs.xfs -f /dev/sdb

mkdir /bigdata
mount /dev/sdb /bigdata

blkid                 # get UUID
vi /etc/fstab         # add entry with UUID and mount point

df -h                 # verify mount

Following these commands ensures the disk is properly formatted, mounted, and persists across reboots.

Reference

xfs_repair command details – https://bbs.qunyingkeji.com/2052/