How Diffy Revolutionizes Regression Testing and Cuts Debugging Effort

Introduction

Regression testing is a painful problem for many engineers. Twitter Diffy offers a new perspective by comparing the output of a stable version and its replica with that of a candidate version, dramatically reducing regression workload.

Background

As software evolves, the number of features grows and systems become more complex. Ensuring that new or modified functionality does not break existing features requires extensive regression testing, which can be costly and often yields few defects.

How Diffy Works

Diffy acts as a proxy that forwards requests to three versions of a system and compares their outputs to draw conclusions.

Candidate version : the version under test, containing new code.

Stable version : the currently deployed version known to work correctly.

Stable‑copy version : a replica of the stable version used to filter out noise.

The workflow is illustrated below:

Diffy identifies the raw differences between candidate and stable outputs, then removes noise by comparing the stable version with its copy. The remaining differences represent true defects.

Simple Example

Installation and basic usage steps:

Install Diffy.

Start the candidate service, stable service, and stable‑copy service.

Run Diffy.

Send requests and view results.

In a local demo, the candidate returns “hello world”, while the stable and its copy return “hello!!!”. Diffy reports a 100% failure rate because the responses differ in value and length.

Real‑World Case: DaMai Platform

DaMai is a product‑data operation platform. Traditional regression required selecting a subset of core functions, which was time‑consuming and incomplete. By introducing Diffy, all online requests were compared against the test environment, achieving full regression coverage and reducing effort.

Key steps:

Deploy Diffy (candidate: test.xxx.com, stable and stable‑copy: xxx.com).

Record all online requests with Charles, rewrite hosts to point to Diffy, update cookies, and replay.

Review Diffy results (e.g., 1,167 requests, 118 differences). Differences are categorized as harmless (e.g., timestamp changes) or true bugs.

Optimization and Lessons Learned

Diffy cannot proxy to test environments directly. A Charles proxy layer was added to forward requests.

Too many requests. Use a recording tool (e.g., Charles) to capture and replay requests instead of sending them manually.

Header differences. Enable excludeHttpHeadersComparison=true to ignore header noise.

Data mismatches between stable and candidate. Diffy works best for gray‑release scenarios where data is identical.

Real‑time interfaces. Minor timing differences may cause false positives.

Production‑only APIs. Avoid invoking APIs that trigger side effects in live environments.

Redirected endpoints. Diffy does not follow redirects; it only returns the original response.

Appendix: Environment Setup

Diffy is written in Scala and maintained on GitHub. To build:

Clone the repository: git clone https://github.com/twitter/Diffy.git Build with sbt: cd Diffy && ./sbt assembly Start candidate, stable, and stable‑copy services.

Run the generated JAR or executable.

Key configuration parameters:

candidate='PC1:8888' – address of the version under test.

master.primary='PC2:8888' – address of the stable version.

master.secondary='PC3:8888' – address of the stable‑copy.

service.protocol='http' – protocol (http or https).

proxy.port=:9990 – Diffy proxy port.

admin.port=:9991 – admin UI port.

http.port=:9999 – UI for viewing differences.

responseMode=primary – which version’s response to return.

allowHttpSideEffects=true – allow POST/PUT/DELETE to be proxied.

excludeHttpHeadersComparison=false – whether to ignore header differences.

After starting Diffy, access http://PC0:9990/admin to view request statistics and http://PC0:9999 to see diff results.

References

https://www.infoq.cn/article/diffy-twitter-open-source-automation-testing-tool

https://blog.csdn.net/liusf1993/article/details/85231239