How to Rescue a Broken CentOS 7 System Caused by Mixed GLIBC Packages

Background

A CentOS 7 server reported GLIBC errors because both CentOS 7 and CentOS 8 packages were present, corrupting the system libraries. Reinstalling the whole environment would be costly, so the author chose to repair it in rescue mode.

Repair Process

Mount the CentOS 7 ISO in rescue mode and reinstall glibc using the ISO's RPM packages.

Enter the rescue environment and mount the original system partition at /mnt/sysimage. chroot /mnt/sysimage Since chroot fails, use rpm --root to reinstall packages directly in the target root.

mount /dev/cdrom /mnt/cdrom
find /mnt/cdrom -name "glibc*.rpm"

Install glibc and glibc-common with --force --nodeps to avoid dependency checks.

rpm --root /mnt/sysimage -ivh --force --nodeps /mnt/cdrom/Packages/glibc-2.17-*.rpm
rpm --root /mnt/sysimage -ivh --force --nodeps /mnt/cdrom/Packages/glibc-common-2.17-*.rpm

Parameters: --force --nodeps – force overwrite, ignore dependencies. --root /mnt/sysimage – target system.

Reinstall glibc without running %pre/%post/%trigger scripts to avoid failures caused by the broken environment.

rpm --root /mnt/sysimage -Uvh --force --nodeps --nopre --nopost --notriggers \
    /mnt/cdrom/Packages/glibc-2.17-*.rpm \
    /mnt/cdrom/Packages/glibc-common-2.17-*.rpm

--nopre --nopost --notriggers = skip scriptlets, preventing "scriptlet failed, exit 127".

Fix critical symlinks for the dynamic linker and libc.

ln -sf /lib64/ld-2.17.so   /mnt/sysimage/lib64/ld-linux-x86-64.so.2
ln -sf /lib64/libc-2.17.so /mnt/sysimage/lib64/libc.so.6

Bind necessary pseudo‑filesystems and chroot into the repaired root.

mkdir -p /mnt/sysimage/mnt/cdrom
mount /dev/sr0 /mnt/sysimage/mnt/cdrom
mount -o bind /dev  /mnt/sysimage/dev
mount -o bind /proc /mnt/sysimage/proc
mount -o bind /sys  /mnt/sysimage/sys
chroot /mnt/sysimage /bin/bash

Create a temporary YUM repo on the ISO and reinstall glibc and related packages.

cat >/etc/yum.repos.d/cdrom.repo <<'EOF'
[cdrom]
name=CentOS7 ISO
baseurl=file:///mnt/cdrom
enabled=1
gpgcheck=0
EOF

yum reinstall -y glibc glibc-common

Identify and remove EL8 packages that polluted the system (e.g., libxcrypt, glibc‑el8, python3), then reinstall the correct EL7 versions.

rpm --root / -qa | grep glibc
rpm -e --nodeps glibc-headers-2.28-164.el8.x86_64
... (other EL8 glibc packages)

rpm -ivh --force --nodeps /mnt/cdrom/Packages/python-2.7*.el7.x86_64.rpm
ln -sf /usr/bin/python2.7 /usr/bin/python

Reinstall PAM, shadow-utils, util‑linux, and authconfig to restore login functionality.

rpm -ivh --force --nodeps /mnt/cdrom/Packages/pam-1.1.8-23.el7.x86_64.rpm
rpm -ivh --force --nodeps /mnt/cdrom/Packages/shadow-utils-4.6.5.el7.x86_64.rpm
rpm -ivh --force --nodeps /mnt/cdrom/Packages/util-linux-2.23.2-65.el7.x86_64.rpm
rpm -ivh --force --nodeps /mnt/cdrom/Packages/authconfig-6.2.8-30.el7.x86_64.rpm

Run authconfig --enableshadow --passalgo=sha512 --update with the restored Python 2.7 interpreter, then refresh the dynamic linker cache.

/usr/bin/python2.7 /sbin/authconfig --enableshadow --passalgo=sha512 --update
/sbin/ldconfig

Final clean‑up: correct remaining symlinks for libcrypt and libnsl, run ldconfig on the target root, and verify login dependencies.

ln -sf /lib64/libcrypt-2.17.so /mnt/sysimage/lib64/libcrypt.so.1
ln -sf /usr/lib64/libnsl-2.17.so /mnt/sysimage/lib64/libnsl.so.1
/sbin/ldconfig -r /mnt/sysimage

readlink -f /lib64/libnsl.so.1
readlink -f /lib64/libcrypt.so.1
ldd /bin/login | grep 'not found' || echo "login deps OK"
getent passwd root

Unmount bind mounts, exit rescue mode, and reboot. The system boots normally and login works.

Conclusion

The root cause was EL8 packages mixed into an EL7 system, corrupting glibc, libcrypt, and libnsl as well as PAM, NSS, util‑linux, and authconfig. By force‑rolling back the glibc family from a matching ISO, fixing critical symlinks, restoring PAM and Python 2.7, and cleaning all EL8 remnants, the server was fully recovered.