Why Does My VPN Hang on MySQL SHOW DATABASES? Uncovering MTU Misconfigurations

1. Environment Topology

All device interfaces have MTU 1450 (20 B IP header + 20 B TCP header + 1410 B TCP payload).

RDS is a JD Cloud database instance with 400 databases.

SH_MariaDB_VM is a Shanghai cloud host running MariaDB with the same data as RDS.

BJ_VPNGW and SH_VPNGW are VPN gateways for Beijing and Shanghai VPCs.

bj_client routes to RDS via the VPN gateway in its own region.

2. Phenomenon Description

bj_client connects to RDS through BJ_VPNGW; the SHOW DATABASES command hangs without response.

bj_client connects to SH_MariaDB_VM through BJ_VPNGW; the SHOW DATABASES command works normally.

3. Troubleshooting Process

bj_client can ping RDS, telnet to port 3306, and connect with the MySQL client.

Packet capture on BJ_VPNGW and SH_VPNGW shows RDS replies reach SH_VPNGW but never arrive at BJ_VPNGW; packets are dropped at SH_VPNGW, likely due to MTU.

Further capture reveals both SH_VPNGW and RDS send ICMP Error messages indicating a next‑hop MTU of 1374, but RDS continues retransmitting packets with total length 1450 because a preceding load balancer does not forward the ICMP messages.

Reducing bj_client’s interface MTU to 1374 resolves the issue.

4. Root Cause

Both bj_client, RDS, and SH_MariaDB_VM use MTU 1450, so during the TCP three‑way handshake the negotiated MSS is 1410 (MTU ‑ 20 B IP header ‑ 20 B TCP header). The DF flag prevents fragmentation. When RDS responds to SHOW DATABASES, each TCP segment is 1410 B. After VPN encapsulation (ESP, UDP, IP), the packet size exceeds the VPN gateway’s egress MTU, causing the gateway to drop the packet. The gateway sends an ICMP Error with MTU = 1374, but the upstream load balancer discards this ICMP, so RDS never reduces its packet size, leading to a drop loop.

5. Solution

Reduce the MTU on the server or client to a value below the path MTU (e.g., the value discovered by ping‑MTU testing).

Configure the VPN gateway to clamp MSS to the path MTU, e.g.,

iptables -t mangle -A FORWARD -o eth0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

.

Ensure ICMP Error messages can reach the RDS instance (e.g., adjust load balancer settings).

6. Additional Information

What is MTU? MTU (Maximum Transmission Unit) limits the payload size at the data‑link layer. Example: MTU = 1450 ⇒ 20 B IP header + 20 B TCP header + 1410 B TCP data.

What is MSS? MSS (Maximum Segment Size) is the largest amount of TCP payload that can be sent in a single segment, negotiated during the TCP handshake (does not include the TCP header).

How to probe Path MTU?

Linux: ping x.x.x.x -s 1422 -M do (1422 B data + 8 B ICMP header = 1450 B total).

Windows: ping x.x.x.x -l 1472 -f.

How to view MTU?

Linux: ifconfig Windows:

netsh interface ipv4 show subinterfaces

How to set MTU?

Linux: ifconfig eth0 mtu xxx Windows:

netsh interface ipv4 set subinterface "WLAN" mtu=1450 store=persistent