Fixing OCP “Invalid VIP” Errors: Why It Happens and Step‑by‑Step Solutions

Background

When accessing certain monitoring data through the OCP master node IP/VIP in a multi‑node OCP architecture, a 401 error may appear, indicating that the VIP (e.g., 10.186.65.251) is invalid.

The log shows the message:

OCP version: 4.3.6-20250815111311

Problem Analysis

Viewing OCP‑registered nodes

docker exec -it ocp bash
cd ocp-server/
log/

tail -f ocp.log | grep '"status":401'

OCP stores node IPs in the ocp.distributed_server table; the VIP is not recorded because this matches product expectations.

Inspecting node IPs

obclient -u${OCP_METADB_USER} -p${OCP_METADB_PASSWORD} -h${OCP_METADB_HOST} -P${OCP_METADB_PORT} -Ac
select * from ocp.distributed_server;

Result shows node IPs 10.186.65.58, 10.186.65.59, 10.186.65.60 and the VIP 10.186.65.251 as a secondary address.

Checking the source IP returned by the kernel

ip route get 10.186.65.59
ip route get 10.186.65.60

Both commands return src 10.186.65.251, meaning the kernel uses the VIP as the source IP, which OCP rejects.

Root Cause

The kernel selects the VIP ( 10.186.65.251) as the source address when node 58 contacts nodes 59/60, while OCP expects a registered node IP (e.g., 10.186.65.58) and therefore returns the 401 error.

Solution

Adjust the source IP used when accessing nodes 59/60 so that the kernel returns the registered node IP 10.186.65.58.

Method 1 – Remove the VIP from the node

ip a del 10.186.65.251/24 dev eth0
ip a show eth0

After deletion, the primary IP becomes 10.186.65.58, and the kernel uses it as the source. Note that the load balancer may briefly re‑attach the VIP, causing short‑lived connection drops.

Method 2 – Add a policy routing rule

# Add a new routing table (ID 100)
cat /etc/iproute2/rt_tables
echo "100 mytable" >> /etc/iproute2/rt_tables

# Route rule for target 10.186.65.59
ip rule add to 10.186.65.59 lookup mytable
ip route add 10.186.65.59 dev eth0 src 10.186.65.58 table mytable

# Route rule for target 10.186.65.60
ip rule add to 10.186.65.60 lookup mytable
ip route add 10.186.65.60 dev eth0 src 10.186.65.58 table mytable

This method is more standard and reliable because it explicitly forces the kernel to use 10.186.65.58 as the source IP for the specified destinations.

Verification

ip route get 10.186.65.59
ip route get 10.186.65.60

Both commands should now return src 10.186.65.58.

Additional Note

Even after fixing the source IP, the error may persist until old connections that still use the VIP are closed. Use the following commands to monitor and clear old connections:

# Find established connections using the VIP
netstat -nap | grep -i ESTABLISHED | grep 251 | grep ':8080'

# Find established connections using the node IP
netstat -nap | grep -i ESTABLISHED | grep 58 | grep ':8080'