How to Use tcpcopy for Real-Time Traffic Replay and Stress Testing

Assume we need to launch an advanced architecture that will not crash for two years. Before launch, we inevitably perform unit tests, functional tests, and stress tests using tools like ab and webbench.

These steps do not reflect real user behavior in production. You might consider a gray‑release, but that could affect some users, contradicting the two‑year no‑downtime promise.

Fortunately, NetEase engineer Wang Bin developed tcpcopy, which can import live traffic for functional and stress testing.

tcpcopy Introduction

tcpcopy

is a request‑copy tool that can duplicate online traffic to a test machine, simulating the production environment in real time without impacting real users, allowing early bug detection and, by amplifying traffic, evaluating system capacity. tcpcopy captures request packets at the online server's IP layer, modifies relevant fields, and uses raw socket output (one of the packet injection techniques) to send them to the test server.

The packets sent to the test server are recognized by the TCP/IP stack; those containing a payload (TCP data) finally reach the upper‑layer application (e.g., nginx). After processing, the response is passed back to the test server's TCP/IP stack.

On the test server, the ip_queue module is enabled and iptables directs response packets to the QUEUE ( ip_queue).

The test server runs an intercept program in user space, which opens a netlink socket to receive packets passed by ip_queue (the response content from the upper‑layer application), judges them, and returns the result to the kernel for dequeuing. By default, the intercept process drops the application response and returns only the IP header to release the TCP connection.

The intercept process can also use the -x (passlist) parameter to avoid dropping responses from specified IP lists. The default drop policy aims to:

1 Reduce outbound bandwidth usage and save costs
2 Avoid affecting the client’s TCP/IP stack
3 Prevent generation of ghost packets on the Internet

tcpcopy Workflow

① A request reaches the online front‑end machine;
② The socket packet is copied at the IP layer and handed to the tcpcopy process;
③ tcpcopy modifies the destination and source addresses and sends the packet to the target test machine;
④ The copied packet arrives at the test machine;
⑤ The test machine’s nginx processes the request and returns a result;
⑥ The result is intercepted and dropped at the IP layer; the intercept process copies the IP header of the result and returns it;
⑦ The IP header is sent back to the online front‑end machine’s tcpcopy process.

Installation and Configuration

tcpcopy has two operating modes:

1 Real‑time packet copying
2 Offline replay using packet capture files (e.g., from tcpdump)

If you choose the real‑time mode, install tcpcopy on both the online server and the test server. For offline mode, only the test server needs tcpcopy, compiled with --enable-offline.

Installation Steps

wget https://github.com/wangbin579/tcpcopy/archive/0.9.0.tar.gz -O tcpcopy-0.9.0.tar.gz --no-check-certificate
tar zxvf tcpcopy-0.9.0.tar.gz
cd tcpcopy-0.9.0
./autogen.sh
./configure --prefix=/usr/local/tcpcopy
make
sudo make install

Test Server Configuration

The test server must use iptables to send response packets to the QUEUE ( ip_queue) so that the intercept process can retrieve them via a netlink socket. Therefore, enable the ip_queue kernel module and start the firewall.

modprobe ip_queue
/etc/init.d/iptables start

Add firewall rules to allow communication between the online server and the test server:

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 36524 -j ACCEPT

Restart iptables: /etc/init.d/iptables restart Then direct outgoing traffic to the QUEUE: iptables -I OUTPUT -p tcp --sport 80 -j QUEUE This sends the response packets to ip_queue, allowing the user‑space intercept process to judge them.

Related Usage

This example assumes both the online and test web services use port 80, and control information uses the default port 36524. Adjust iptables rules if you use different ports.

Offline Mode

1) Capture traffic on the online server:

tcpdump -i eth0 tcp and port 80 -s 0 -w online.pcap

2) Copy the capture file to the test server.

3) Replay the traffic on the test server:

cd /usr/local/tcpcopy/bin
sudo ./intercept
sudo ./tcpcopy -i /path/online.pcap -x 80-10.16.12.11:80

# General form
tcpcopy -i <capture_file> -x <port>-<test_server_ip>:<port>

Online Real‑Time Copy

1) Start the intercept process on the test server:

cd /usr/local/tcpcopy/bin
sudo ./intercept

2) On the online server, copy traffic to the test server:

cd /usr/local/tcpcopy/bin
sudo ./tcpcopy -x 80-10.16.12.11:80 -c 10.16.12.12

# General form
tcpcopy -x <server_port>-<test_server_ip>:<test_server_port> -c <local_server_ip>

References:

https://github.com/wangbin579/tcpcopy

http://www.searchtb.com/2012/05/using-tcpcopy-to-simulate-traffic.html

http://hi.baidu.com/yacker/item/e6bd5b287fe5a3f150fd8731

http://blog.yam.com/hn12303158/article/35207136