How a Single Space in ifconfig Crashed an Oracle RAC Cluster – Ops Lessons

System operations demand meticulous attention; a single character mistake can cause catastrophic failures.

Fault Symptoms

A 10.2.0.4 Oracle RAC for Solaris 10 environment unexpectedly restarted instances on both nodes.

Fault Process

At around 15:00 the two nodes rebooted; one node's instance failed to start automatically. Log inspection revealed ORA‑27504 IPC errors on both nodes before the reboot.

Error Messages

ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation: if_not_found failed WITH STATUS: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 192.168.168.3 NOT found. CHECK output FROM ifconfig command

The messages clearly indicate a missing IP address, prompting a check of the ifconfig output.

IPC Timeout and Instance Eviction

Wed Apr 10 15:08:13 2013 – network interface with IP 192.168.168.3 no longer operational IPC Send timeout detected. Sender: ospid 25748, Receiver: inst 2 binc 430164 ospid 11890 Wed Apr 10 15:16:40 2013 – Waiting FOR instances TO leave: 2

The root cause was a typo when checking IPv6 addresses: the command ifconfig –a 6 (with a space) was entered instead of ifconfig –a6, causing the system to set all IPs to 0.0.0.0.

Lesson

Privileged users, including DBAs and root, must verify every command character to avoid cluster‑wide outages.

ifconfig Command Reference

The ifconfig utility configures and displays network interface parameters on Linux.

Syntax ifconfig [interface] [options] Common Parameters add<address> – add an IPv6 address del<address> – delete an IPv6 address down – deactivate the interface up – activate the interface hw ether <MAC> – set MAC address mtu <bytes> – set maximum transmission unit netmask <mask> – set subnet mask broadcast <addr> – set broadcast address

Examples

Activate eth0: ifconfig eth0 up Deactivate eth0: ifconfig eth0 down Set IPv4 address:

ifconfig eth0 192.168.2.10 netmask 255.255.255.0 broadcast 192.168.2.255

Configure IPv6 address: ifconfig eth0 add 33ffe:3240:800:1005::2/64 Delete IPv6 address: ifconfig eth0 del 33ffe:3240:800:1005::2/64 Enable ARP: ifconfig eth0 arp Disable ARP:

ifconfig eth0 -arp