What Triggered These Real‑World System Crashes? 13 Post‑Mortem Lessons

01. AIX NTP misconfiguration caused multiple cluster crashes

A friend reported that three Oracle RAC clusters on AIX machines rebooted simultaneously after a hardware relocation. Investigation revealed all clusters shared the same NTP server, but one used xntpd while the others used ntpdate via cron. The ntpdate jobs caused large time jumps, which made the cssd process trigger a system reboot. Lesson: Prefer the xntpd service for time synchronization instead of periodic ntpdate calls.

02. Backup appliance CDP driver caused a crash

During testing of an AIX backup appliance, the CDP driver was left installed after the client was removed. Upon reboot the system failed to start. The vendor confirmed that the CDP driver must be removed before uninstalling the client.

03. LVM mirror expansion error led to data loss

In a dual‑node, dual‑storage HA setup, expanding a filesystem by adding disks directly to the VG caused data to be unevenly distributed across the two storage arrays. When one storage failed, the system lost data integrity. Lesson: When using LVM mirrors, expand the logical volume first, then the filesystem.

04. HACMP node‑ID duplication caused cluster halt

Three PowerHA XD clusters shared identical RSCT node UUIDs after an alt_disk_copy without the -B -C -O options. The duplicate IDs caused quorum loss and a complete halt. The fix involved stopping HA services, reinstalling the RSCT node configuration, and rebooting all nodes.

05. Power 570/595 crash due to improper CDP driver removal

After uninstalling the backup client but leaving the CDP driver, the Power 595 failed to boot. The vendor required the CDP driver to be removed first.

06. ERP backup triggered HACMP crash

During a backup window, the haemd daemon repeatedly restarted, causing the Oracle database to stop. The issue stemmed from excessive I/O and insufficient filesystem cache, which was mitigated by adjusting Maxpout and Minpout parameters.

07. WebLogic memory‑leak crash investigation

Repeated out‑of‑memory errors were traced to non‑heap memory exhaustion. Adjusting PermSize in setDomainEnv.sh had no effect because JAVA_VENDOR was set to N/A. The final fix set a proper JAVA_VENDOR and added explicit memory arguments ( -Xms2048m -Xmx2048m -XX:PermSize=1024m).

08. P550/P570 HA crash and data loss

Power failure left both UPS units partially powered, causing both P550 nodes to shut down. After hardware replacement and manual IP aliasing, the HA cluster was restored, though some /orafile data was lost and later recovered from backup.

09. AIX 6100‑06‑06 bug causing kernel panic

The netstat -f unix command triggered a kernel panic due to a file‑lock bug (IV09793). The recommended fix is to apply the bos.mp64 patch or upgrade to level 6100‑06‑12‑1339 (SP12).

10. PowerHA node‑ID conflict during IP switch

When all IP networks were lost but a non‑IP network remained, PowerHA 6 dumped core (IV55293). Upgrading the rsct fileset resolved the issue.

11. Power595 crash caused by I/O cabinet power loss

During a routine I/O cabinet power‑swap, an unexpected power drop caused the Power595 to crash. Replacing the I/O DCA resolved the problem.

12. X86 server crash due to faulty optical drive

An IBM X3650 running SUSE 9 hung because a defective CD/DVD drive caused kernel panics. Replacing the drive restored stability.

13. Miscellaneous hardware‑related crashes

Additional incidents include UPS failures, firmware errors, and component replacements that led to temporary outages but were resolved through hardware swaps and firmware updates.