How tcpcopy Evolved: From Simple Request Copying to Scalable Traffic Replication

In the field of server request replay, research typically focuses on offline replay, while online real‑time replication receives far less attention.

Request real‑time replication can be divided into two categories: application‑layer request copying and lower‑layer packet‑level request copying.

Application‑layer copying is simple to implement but has several drawbacks, such as consuming valuable connection resources, coupling test traffic with production systems, difficulty handling high‑load scenarios, and limited control over network latency.

Packet‑level copying avoids traversing the full protocol stack by capturing packets at the data‑link or IP layer, reducing impact on the online system.

First Architecture

The initial tcpcopy design captures request packets at the data‑link layer via a pcap interface and sends them from the IP layer. Response packets return to the online machine and are also captured at the data‑link layer, where they are typically discarded.

Advantages:

Simple and straightforward.

Suitable for smoke testing.

Produces relatively realistic test results.

Disadvantages:

Higher impact on the online system because all response packets are returned.

Limited to the same subnet.

Cannot fully utilize multiple online streams for external services.

Internal network applications are heavily constrained.

Second Architecture

The current open‑source design captures request packets at the IP layer and uses an intercept program on the test server to capture response packets, forwarding necessary TCP/IP header information back to tcpcopy.

Advantages:

Supports copying traffic from multiple online sources.

Reduced impact on the online machine, as only header information is returned.

Disadvantages:

More complex than the first architecture.

Performance limited by ipqueue or nfqueue.

Intercept scalability is limited; it cannot easily handle multi‑process response capture.

Intercept consumes resources on the test server and may affect results under heavy load.

Cannot fully test the data‑link layer.

Operationally inconvenient.

Third Architecture

The latest design offloads the intercept function to an independent assistant server and captures response packets at the data‑link layer, dramatically reducing interference with the test server.

Advantages:

More realistic traffic replay.

Higher scalability, suitable for extreme concurrency.

No reliance on ipqueue or nfqueue.

Minimal performance impact on the test server.

Operations are easier on the test server and the system is more resilient.

Disadvantages:

Increased operational difficulty.

Requires additional machines.

Needs deeper knowledge of TCP, routing, and pcap filtering.

The assistant server must reside in the same subnet as the test server.

Only the second and third architectures are publicly available; tcpcopy defaults to the second, while the third can be adopted when conditions permit.

For version 0.9, migration guidance is available at the referenced blog; version 1.0 and later use the new architecture by default.

To minimize impact on production systems during request replication, high‑performance bypass mechanisms (e.g., traffic mirroring with destination address rewriting) and the third architecture are recommended.

Source: https://blog.csdn.net/wangbin579/article/details/8949315