Master Linux Network Diagnostics: Essential Commands and Tips for High‑Performance Servers

Resource estimation for large numbers of connections

Each TCP connection consumes a file descriptor and about 15‑20 KB of socket memory. One million concurrent connections therefore need roughly 20 GB of RAM for sockets and about 1 Gbps of bandwidth for a 1 KB broadcast.

Viewing current system connections

Use netstat with awk to count connections by state:

# netstat -antp | awk '{a[$6]++} END {for (x in a) print x, a[x]}'
LISTEN 41
CLOSE_WAIT 24
ESTABLISHED 150
Foreign 1
TIME_WAIT 92

For high‑connection servers, ss is faster and more lightweight:

# ss -s
Total: 191 (kernel 220)
TCP: 5056 (estab 42, closed 5000, orphaned 3, synrecv 0, timewait 5000/0), ports 3469
...

netstat

belongs to the legacy net-tools suite; ss is part of iproute2.

Common ss usage

Show listening TCP sockets : ss -atr (or ss -atn for numeric IPs)

Show all sockets : ss -alt Find process listening on a port : ss -ltp | grep 444 Show UDP sockets : ss -u -a Filter by destination IP : ss dst 10.66.224.130, ss dst 10.66.224.130:http, etc.

Show all HTTP connections : ss dport = :http Top 10 remote IPs by connection count :

netstat -antp | awk '{print $4}' | cut -d ':' -f1 | sort | uniq -c | sort -nr | head -n 10

Recv‑Q and Send‑Q

In LISTEN state, Recv‑Q shows connections waiting for accept() and Send‑Q shows the backlog size. In ESTABLISHED state, Recv‑Q indicates bytes pending read by the application, and Send‑Q indicates bytes awaiting acknowledgment.

Monitoring network traffic

Per‑second traffic statistics with sar: sar -n DEV 1 Or watch /proc/net/dev directly: watch cat /proc/net/dev Identify the host consuming the most bandwidth with iftop (interactive display).

Packet capture with tcpdump

Capture packets for debugging and analyze with Wireshark:

tcpdump -i eth0 -nn -s0 -v port 80

-i

interface -n no DNS resolution -nn keep ports numeric -s 0 capture full packet -v verbose (add more v for more detail) -A print ASCII payload -X print hex and ASCII -w file.pcap write to file

Examples:

tcpdump -A -s0 port 80
tcpdump -i eth0 host 10.10.1.1
tcpdump -i eth0 dst 10.10.1.20
tcpdump -s 0 -v -n -l | egrep -i "POST /|GET /|Host:"

More examples are listed at https://hackertarget.com/tcpdump-examples/

HTTP‑level capture tools

Proxy‑style capture and replay tools include Burp Suite (cross‑platform), Fiddler (Windows) and Charles (macOS).

Traffic replication

To replay production HTTP traffic in a test environment, tools such as Gor, TCPReplay and TCPCopy can be used.

Connection‑count problems

Large numbers of TIME_WAIT and CLOSE_WAIT sockets often indicate kernel or application issues.

TIME_WAIT

Occurs on the side that actively closes a TCP connection. High counts can be mitigated by tuning kernel parameters:

# Increase bucket limit
net.ipv4.tcp_max_tw_buckets = 50000
# Enable fast reuse of TIME_WAIT sockets
net.ipv4.tcp_tw_reuse = 1
# Enable fast recycle (kernel 4.12+)
net.ipv4.tcp_tw_recycle = 1
# Reduce FIN timeout (default 60 s)
net.ipv4.tcp_fin_timeout = 10

Apply immediately with sysctl -w net.ipv4.tcp_tw_reuse=1 or persist in /etc/sysctl.conf and reload via sysctl -p.

CLOSE_WAIT

Indicates that the remote side closed the connection but the local application failed to close its socket, typically due to missing close() or abort() calls.

Other frequently used commands

File transfer and download

# Resume download
wget -c $url
# Mirror entire site
wget -r -p -np -k $url
# Simple HTTP GET
curl -XGET $url
# Secure copy
scp
sftp
# Data synchronization
rsync

Network diagnostics

# Ping
ping google.com
# Trace route
tracepath google.com
# DNS lookup
dig google.com
nslookup google.com
# Port scanning
nmap
# Bandwidth test
iperf
# System monitoring
nmon

Interface management

# Bring interface down
ifdown
# Bring interface up
ifup
# Advanced NIC control
ethtool

Load testing

wrk
ab
webbench
http_load

General utilities

# Remote login
telnet
ssh
nc
# Firewall rules
iptables -L

Conclusion

Most of the commands described are not installed by default and may require yum or apt installation. For deeper networking knowledge, consult “TCP/IP Illustrated, Volume 1” and build small Netty applications.