How to Rescue MySQL Data from a Crashed Linux VM Using LVM

Overview

If a Linux system crashes and cannot be rescued with the standard rescue mode, the MySQL databases on that machine may still be recoverable. This article demonstrates a practical experiment to answer that question.

Environment

Two VirtualBox VMs are prepared:

IP 192.168.56.81 – the original machine running MySQL (later simulated to fail).

IP 192.168.56.71 – a fresh machine that will host the recovered data.

Images of the original file system and MySQL data are shown below:

Experiment Steps

Delete the /boot directory on VM 81 and force a reboot. The system fails to start, reporting error: file ‘/grup2/i386-pc/normal.mod’ not found.

Power off VM 81 and force exit.

Attach the virtual disk of VM 81 to VM 71 and boot VM 71.

Use lsblk to identify the newly attached disk ( sdb).

Attempt to mount /dev/sdb2; the kernel reports unknown filesystem type ‘LVM2_member’ because the partition is a physical volume.

Activate the LVM volume group with vgchange -ay /dev/vg (replace vg with the actual VG name). After activation, the logical volume status changes to “available”.

Mount the logical volume, e.g.:

mkdir /newmnt
mount /dev/vg/lvroot /newmnt

and verify that the original file hierarchy is visible.

Copy the MySQL files to the new machine:

Install the same MySQL version on VM 71.

Inspect the original my.cnf to locate basedir, datadir, log_bin, etc.

Create the required directories on the new host and set ownership to mysql:mysql.

mkdir -p /opt/mydata/data
mkdir -p /opt/mydata/tmp
mkdir -p /opt/mydata/log/binlog
chown -R mysql:mysql /opt/mydata

Copy configuration and data files:

cp -r /newmnt/etc/my.cnf /etc
cp -r /newmnt/opt/mydata/data /opt/mydata/
cp -r /newmnt/opt/mydata/log/binlog /opt/mydata/log/

MySQL Restart

Attempt to start MySQL: service mysql start The service fails because the error log file does not exist. Create it and retry:

touch /opt/mydata/log/error.log
chown -R mysql:mysql /opt/mydata/log/error.log
service mysql start

MySQL starts successfully and the database tables are restored, as confirmed by listing the recovered files.

Conclusion

If the database files have not been deleted, even a completely unbootable system can be salvaged by detaching its disk, activating LVM volumes, and copying the data to a new host.

Regular backups for production, testing, and development environments are essential to avoid such recovery procedures.