How Hunter Automates Web Security Scanning in CI/CD Pipelines

Background

In many companies, quality testing engineers ensure functional quality but often lack security expertise, leading to missed security risks before a service goes live. Traditional manual security checks are slow and reactive, prompting the need for automated security scanning within the release pipeline.

Problem and Challenges

Security teams must discover vulnerabilities early, yet limited manpower makes it impossible to manually test every release. Common web vulnerability scanners use headless browsers to crawl sites and test interfaces, but they cannot handle interactive pages and are inefficient because they re‑crawl the entire site on each iteration.

Capturing traffic generated by quality‑testing engineers offers a solution, but traditional network‑proxy methods have drawbacks. Using a browser extension to capture traffic provides better client‑side debugging, avoids double‑proxy configuration, enables interactive notifications, and can cooperate with server‑side storage‑XSS detection.

Hunter Architecture

The system, named "Hunter," consists of a browser plugin (client), a RESTful API, a RabbitMQ message queue, a distributed detection engine, and a visualization platform. MySQL stores persistent data, while RabbitMQ’s fan‑out exchange distributes tasks to multiple queues for SQL‑injection and XSS detection.

Implementation Details

Client (Browser Extension)

When a user creates a new task, the extension displays a user agreement dialog using a content script that injects HTML into the page. The user configures regex rules for domain capture, task name, and notification email.

Traffic Capture

In background.js, the extension registers chrome.webRequest.onBeforeRequest and chrome.webRequest.onBeforeSendHeaders to intercept network requests. Listeners are removed when the task stops, and communication with the content script occurs via chrome.runtime.sendMessage.

RESTful API

The captured requests are packaged and sent to the API, which validates them against a whitelist (e.g., test‑environment IP ranges) before pushing them to RabbitMQ for analysis.

Distributed Detection Engine

Workers consume messages from RabbitMQ, performing multi‑POC checks per URL. SQL‑injection and XSS checks run in separate queues using a fan‑out exchange. The engine can also execute POC plugins for flexible vulnerability expansion.

Storage‑XSS Detection

Reflection/DOM XSS is detected with a headless Chrome backend. For stored XSS, the client monitors the page for injected payloads (identified by a request ID) and reports findings to the API, enabling coverage of both XSS types.

Visualization Platform

After scanning, results are stored and displayed in a web dashboard, allowing users to view historical tasks, detailed vulnerability reports, and statistical charts.

Usage Practice

Users configure regex capture rules, email recipients, and task names when creating a new scan. After the quality‑testing engineer performs normal interface testing, the captured traffic is processed automatically. Scan records and vulnerability details can be reviewed in the dashboard.

Conclusion and Outlook

Hunter now detects most common web vulnerabilities except privilege‑escalation issues, which are under development. Future work includes adding more scanning plugins, enhancing online operations, and integrating internal permission systems for privilege‑escalation detection. The goal is to eliminate security risks before deployment, and the team plans to open‑source Hunter to benefit more enterprises.