How a Single Space in ifconfig Crashed an Oracle RAC Cluster – Ops Lessons

System operations demand meticulous attention; a single character mistake can cause catastrophic failures.

Fault Symptoms

A 10.2.0.4 Oracle RAC for Solaris 10 environment unexpectedly restarted instances on both nodes.

Fault Process

At around 15:00 the two nodes rebooted; one node's instance failed to start automatically. Log inspection revealed ORA‑27504 IPC errors on both nodes before the reboot.

Error Messages

ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation: if_not_found failed WITH STATUS: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 192.168.168.3 NOT found. CHECK output FROM ifconfig command

The messages clearly indicate a missing IP address, prompting a check of the

ifconfig

output.

IPC Timeout and Instance Eviction

Wed Apr 10 15:08:13 2013 – network interface with IP 192.168.168.3 no longer operational IPC Send timeout detected. Sender: ospid 25748, Receiver: inst 2 binc 430164 ospid 11890 Wed Apr 10 15:16:40 2013 – Waiting FOR instances TO leave: 2

The root cause was a typo when checking IPv6 addresses: the command

ifconfig –a 6

(with a space) was entered instead of

ifconfig –a6

, causing the system to set all IPs to 0.0.0.0.

Lesson

Privileged users, including DBAs and root, must verify every command character to avoid cluster‑wide outages.

ifconfig Command Reference

The

ifconfig

utility configures and displays network interface parameters on Linux.

Syntax

ifconfig [interface] [options]

Common Parameters

add<address>

– add an IPv6 address

del<address>

– delete an IPv6 address

down

– deactivate the interface

up

– activate the interface

hw ether <MAC>

– set MAC address

mtu <bytes>

– set maximum transmission unit

netmask <mask>

– set subnet mask

broadcast <addr>

– set broadcast address

Examples

Activate eth0:

ifconfig eth0 up

Deactivate eth0:

ifconfig eth0 down

Set IPv4 address:

ifconfig eth0 192.168.2.10 netmask 255.255.255.0 broadcast 192.168.2.255

Configure IPv6 address:

ifconfig eth0 add 33ffe:3240:800:1005::2/64

Delete IPv6 address:

ifconfig eth0 del 33ffe:3240:800:1005::2/64

Enable ARP:

ifconfig eth0 arp

Disable ARP:

ifconfig eth0 -arp