Why Oracle RAC Failed to Failover: A Storage Link Diagnosis

Yesterday a colleague reported a recurring RAC issue on an Oracle 11.0.2.4 two‑node cluster where the CRSD service crashed and the cluster failed to failover automatically.

Log analysis showed that the CRSD services on both nodes rebooted within one second of each other, and the voting file /dev/raw/raw1 was reported as non‑functional after a short timeout (approximately 99 seconds). This simultaneous restart indicated the problem was not inherent to RAC itself.

Detailed node logs:

Node 1 at 23:25:38 reported

CRS-1615: No I/O has completed after 50% of the maximum interval. Voting file /dev/raw/raw1 will be considered not functional in 99220 milliseconds

, followed by a CRSD reboot and an exit message Could not init OCR, code: 26.

Node 2 showed a similar message at 23:25:36 with a timeout of 99760 milliseconds, then the same reboot and exit code.

OS kernel logs revealed loss of connectivity on the remote fibre channel port (WWN = 50:0b:34:20:0f:f4:f8:08) for both nodes, confirming a storage link failure.

The issue was traced to the storage subsystem (a domestic product from Hongshan). After contacting the storage vendor, the link problem was addressed.

Key takeaway: When RAC exhibits abnormal behavior, investigators must examine both software and hardware layers. Storage link failures can render the voting file unavailable, causing CRSD crashes and preventing automatic failover.