How DoGNAVY Ranked #3 Globally in AI Security Using a Single Open‑Source Model

CyberGym Ranking and Core Question

In early August, the international AI‑security benchmark CyberGym updated its leaderboard, placing DoGNAVY at 90.8% verification rate, third globally, just one task behind the second place. The ranking prompts a deeper look at how a single open‑source base model can be organized into a system that traverses large codebases, performs hours‑long analysis, and repeatedly constructs and validates PoCs.

What CyberGym Measures

CyberGym Level 1 supplies a vulnerability description, an unpatched source‑code tree, and minimal metadata, testing whether a system can generate an executable candidate PoC. Success requires the candidate to crash the vulnerable version while the patched version remains stable; simple exits, assertion failures, or unrelated crashes do not count.

Out of 1,507 tasks, DoGNAVY generated crashing inputs for 1,453 tasks, with 1,369 passing the primary differential metric, yielding a 90.84% verification rate. Seventy‑nine inputs crashed both versions and were excluded, five were marked “verified but OOM,” and 54 tasks had no candidate PoC.

DoGNAVY’s Multi‑Agent Architecture

DoGNAVY is described as a multi‑agent system for vulnerability reproduction that uses GLM‑5.2 as the sole open‑source foundation model. The model handles reasoning, hypothesis formation, and tool selection, while the system layer manages task decomposition, state retention, execution environment, and verification boundaries.

Agents start from the program’s real entry point, trace the vulnerability path, record parsing stages, required branch and state conditions, and input constraints, then construct candidate PoCs. After execution on the vulnerable version, coverage, error locations, crash types, and stability inform the next analysis round.

An independent review agent checks whether the PoC reaches the expected code path, corresponds to the target vulnerability, and can be stably reproduced. Review results guide the decision to submit the candidate for dual‑version verification, reducing false positives.

The workflow is iterative: if execution shows that input constraints are invalid, the system returns to PoC construction; if the review finds the crash unrelated, it reverts to reachability analysis and revisits earlier assumptions.

Reachability Analysis and Dynamic‑Static Loop

Static analysis proposes vulnerability paths and input constraints; dynamic analysis validates these hypotheses. Each execution feeds back coverage, error location, and crash information, which updates subsequent source analysis or PoC construction. This evidence‑driven loop is summarized as:

Path hypothesis → Input constraints → Candidate PoC → Execution feedback → Updated path or input

Memory is scoped per task: cross‑task memory is disabled to test generalization on unseen open‑source projects, while intra‑task compressed memory retains analyzed paths, confirmed inputs, failed attempts, and unresolved hypotheses.

Three‑Layer Design

Layer 1 – GLM‑5.2: Provides the foundational reasoning capability, influencing the agent’s ability to understand code context, trace call chains, and adjust analysis based on feedback.

Layer 2 – DARKNAVY Experience: Transforms frontline vulnerability research into executable workflows, enabling the system to reconstruct call relationships, verify real reachability, and iteratively refine inputs using runtime evidence.

Layer 3 – AgentDoG: Supplies the core agent execution and diagnostic engine, tracking tool calls, environment feedback, and intermediate actions to pinpoint risk sources, failure reasons, and deviations from expected behavior.

AgentDoG 1.5 Extensions

AgentDoG 1.5 extends safety assessment from final replies to full execution traces, diagnosing risks from user input, tool descriptions, environment feedback, and permission checks. It trains a lightweight trajectory‑safety model for fine‑grained evaluation, filters SFT data during training, and offers a pre‑reply online shield that audits complete traces before final output.

The model also provides structured explanations of risk origin, failure mode, and potential impact, enhancing transparency for security decisions.

Comparison with Other Leaders

Microsoft’s MDASH and Wiz Atlas rely on multiple proprietary models, whereas DoGNAVY achieves comparable performance with a single open‑source model (GLM‑5.2) combined with multi‑agent orchestration and expert security workflows. This demonstrates that open‑source foundations, when paired with robust system engineering and domain expertise, can compete at the highest level of AI‑security benchmarks.

Implications

The result validates an alternative path: leveraging open‑source models, agent system engineering, and real‑world attack‑defense experience to tackle large‑scale, high‑difficulty vulnerability reproduction tasks, highlighting the growing capability of Chinese AI‑security research.