How to Combine Proxies and Scanners to Cut Web Vulnerability False Positives

01 Design

Web security scanning aims to discover vulnerabilities early by crawling target systems, but crawlers often miss deep URLs, leading to false positives and false negatives. Manual testing is accurate but labor‑intensive. The proposed solution captures all real requests via a proxy, stores them in a central database, and feeds them to various scanners, combining speed with precision.

The system consists of the following modules:

Client: Browser or mobile app that initiates requests.

Proxy: Captures every request from the client (e.g., Burp, load balancer).

Parser: Parses captured data and inserts it into a request database.

Request Database: Stores all captured requests and configuration for scanners.

Scanner: Vulnerability detection engines (custom hackUtils, SQLMap, Burp Scanner, WVS, OWASP ZAP, etc.).

Application System: The target web or app under test.

The architecture allows multiple proxies and scanners to operate independently while sharing the same request database, enabling scalable and flexible vulnerability detection.

02 Practice

A demo was built where Burp is configured as a proxy on a browser or mobile device. As users navigate the application, Burp captures all request data—including headers, cookies, methods, and payloads—parses it, stores it in the central database, and distributes it to multiple scanning engines for repeat detection of all controllable input points.

The author also provides a Python requests wrapper (HackRequests) that supports custom cookies, headers, GET/POST requests, and PhantomJS rendering for anti‑scraping and DOM‑based XSS detection. Repository: https://github.com/brianwrf/HackRequests

03 Reflection

Testing on DVWA and WebGoat shows the approach is effective, but several reasons justify rebuilding the solution:

Strong system coupling hampers extensibility.

HTTPS traffic capture is not well supported.

Not all controllable input points (e.g., cookies, User‑Agent, Referer) are inspected.

Lack of DOM rendering leads to missed DOM‑based XSS.

No distributed deployment capability to leverage parallel processing.

Insufficient repeat detection to fully simulate user requests.

Open issues include protecting real user data when mirroring traffic at the front‑end and handling sensitive fields with exceptions.