How Securing MySQL with TCP Wrappers Can Cause an Outage
This article examines a MySQL outage triggered by TCP wrappers, detailing how high thread counts, DNS latency, TIME_WAIT socket exhaustion, and frequent reads of /etc/hosts.allow and /etc/hosts.deny can block connections, and provides troubleshooting steps and mitigation recommendations.
Protecting MySQL is challenging; a complex setup can lead to hard‑to‑diagnose failures. In this case, when active threads exceeded a threshold, MySQL became unresponsive and logs showed repeated "Got an error writing communication packets" messages.
Initial remote testing used a simple loop to verify whether the issue was random, network‑related, or internal to mysqld:
for i in {1..100}; do mysql -h 10.0.2.14 -Bsse "show status like '%uptime';"; doneThe loop confirmed that remote TCP connections sometimes hung, while local socket connections remained responsive, indicating a network‑level block.
Further investigation with netstat -a -t revealed many connections in TIME_WAIT state, suggesting that TCP sessions were being exhausted:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 db4-atsaloux:42000 0.0.0.0:* LISTEN
... (many TIME_WAIT entries) ...Attempts to adjust kernel parameters such as ip_local_port_range and tcp_tw_reuse did not resolve the issue. Packet capture with tcpdump dst port 3306 or src port 3306 showed heavy DNS traffic, and the MySQL variable skip_name_resolve was enabled, so DNS should not have been consulted.
Strace of the mysqld process showed it repeatedly opened and read /etc/hosts.allow and /etc/hosts.deny :
strace -e open,read -p$(pidof mysqld) read(51, "# /etc/hosts.allow: list of host...", 4096) = 464These files are used by TCP wrappers; each new connection triggers an ACL check, which can involve DNS lookups. Slow or failing DNS responses caused the connections to stall or be discarded.
The root cause was therefore the interaction between TCP wrappers and problematic DNS configuration. Even with skip_name_resolve=ON , MySQL still performed DNS lookups via the wrappers.
Recommendations include:
Avoid using TCP wrappers unless necessary, and ensure DNS responses are fast.
Add client hosts to /etc/hosts to bypass DNS lookups.
Configure multiple reliable DNS servers in /etc/resolv.conf and consider caching.
Verify that mysqld is linked against libwrap (e.g., ldd /usr/sbin/mysqld | grep libwrap ) and, if so, review wrapper settings.
For further reading, see MySQL’s “Communication Errors and Aborted Connections” documentation and related Percona blog posts.
Aikesheng Open Source Community
The Aikesheng Open Source Community provides stable, enterprise‑grade MySQL open‑source tools and services, releases a premium open‑source component each year (1024), and continuously operates and maintains them.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.