Can AI Discover Real Vulnerabilities? Researchers Embed Real Bugs into Model Parameters

Problem and Existing Gaps

Current large‑language‑model (LLM) security tools can answer questions but cannot perform the multi‑minute, iterative workflow of real vulnerability research, which requires code search, compilation, execution, analysis, and correction. Open‑source CVE records often provide only brief descriptions, patches, and crash logs, lacking the full build environment, affected and fixed code versions, and automated verdict mechanisms needed for training autonomous agents.

CyberFactory and OpenAegis

A joint team from Beihang University, ELLIS, SMU, and IQuest Research proposes CyberFactory , a unified, reproducible pipeline that reconstructs raw vulnerability data into runnable, automatically‑verifiable tasks and generates agent trajectories. Using this pipeline they train an open‑source security LLM called OpenAegis . The focus is not merely to collect static challenges but to close the entire chain from real‑world bugs to agent‑level training data.

Two Key Breakthroughs

Reconstructing the full task pipeline ("building the problem") before training.

Generating scalable, high‑quality agent trajectories that encode the step‑by‑step analysis process.

From Open‑source CVEs to Trainable Tasks

CyberFactory processes CVE entries through three steps:

Rebuild pre‑ and post‑fix environments: Source repositories are fetched, the fixing commit is located, and both vulnerable and patched versions are compiled into runnable containers, enabling the same input to be executed before and after the fix.

Validate solvability and control difficulty: Runtime evidence is cross‑checked against known vulnerability types, crash reports, and locations. Redundant entries are removed, and any verification‑stage hints are excluded from the training data.

Generate task specifications for agents: For entries with clear fix commits, the relevant information is kept; for vague entries, the system synthesizes a description using available evidence while withholding extra clues that could leak answers.

These steps turn scattered CVE artifacts into concrete, executable security tasks.

Generating High‑Quality Agent Trajectories

To avoid agents wasting time on irrelevant trial‑and‑error, CyberFactory defines a reusable vulnerability‑analysis Skill that outlines a generic workflow independent of any specific task:

Inspect the target program and its build constraints.

Select appropriate analysis and testing methods to explore candidate inputs.

Validate gathered evidence and adjust the method upon failure.

The Skill is applied only during data and trajectory generation; it is not provided to the model at inference time. Verified trajectories are then used for supervised fine‑tuning, transferring the procedural knowledge into model parameters.

OpenAegis Model and Evaluation

OpenAegis (≈ 397 B‑parameter) is trained on three complementary tasks:

Vulnerability‑reproduction input generation.

Patch generation that removes the security issue while preserving functionality.

Security‑focused question answering.

All models are evaluated in the CyberGym benchmark with a one‑hour time limit per task. Results (shown in the figure below) indicate that OpenAegis achieves a 58.1% pass rate, up from 29.6% for the Qwen 3.5 base model—a 28.5‑point absolute gain. It also outperforms GLM 5.2 by 14.8 points and Kimi K2.7 by 6.4 points, despite using the same execution scaffold, tools, and verdict logic.

Long‑Running Task Context Management

Vulnerability reproduction often requires dozens of tool interactions, exhausting the model’s context window. CyberFactory compresses confirmed evidence, failed attempts, pending hypotheses, build state, and next actions into a compact “work state” once 90% of the context is used, allowing the agent to continue without losing critical information.

In context‑management experiments, this strategy yields an overall 58.1% pass rate, with 48.7% success on tasks involving >40 tool calls, and reduces context exhaustion from 18.7% to 7.0%. Compared with retaining the full history, long‑task pass rate improves by 8.5 points and context exhaustion drops by 11.7 points.

Skill Injection and Model Internalization

When the vulnerability‑analysis Skill is directly injected into GLM 5.2, the model runs each task five times (15 min per run) instead of once (60 min). Pass@1 rises from 43.3% to 46.5%, indicating higher trajectory‑generation efficiency, though the comparison is not compute‑equal.

Skill injection also shifts GLM 5.2’s behavior: trajectories following the Skill increase from 2.6% to 99.7%, showing that the model can reliably adopt the prescribed workflow. However, reliance on explicit Skill prompts may cause over‑dependence on domain priors.

OpenAegis, which does **not** receive the Skill at inference, still exhibits a dramatic increase in Skill‑like behavior (0.6% → 85.2% of trajectories) and higher counts of exploratory and verification actions, demonstrating that the Skill was successfully internalized during training.

Further analysis shows OpenAegis makes more terminal calls (70.1% → 89.9%), increases operations per call (2.7 → 5.5), and raises the proportion of successful single‑submission trajectories (37.9% → 48.2%). It also performs far more AddressSanitizer compilations (155 → 1 795) while reducing redundant submissions, indicating more disciplined evidence collection and result submission.

Remaining Challenges

CyberFactory is not yet universal. Open‑source vulnerability material varies in completeness, limiting task reconstruction due to missing versions, dependencies, or build scripts. Fuzzing does not suit all bugs; some tasks still benefit from manual input generation. Moreover, quantitative results currently focus on vulnerability‑reproduction; patching and security‑QA tasks lack equally rigorous evaluation.

Conclusion

The work demonstrates a reproducible path for building security‑focused LLMs: reconstruct tasks from open‑source bugs, generate high‑quality agent trajectories with a domain‑specific Skill, and fine‑tune a model to internalize the procedural knowledge. This closes a long‑standing gap in cybersecurity LLM research—moving beyond static question‑answer pairs to teaching models how to *act* and solve real security problems.

Paper: "CyberFactory: Scaling Cyber Security Capabilities with Instances from the Wild"

Resources: https://huggingface.co/collections/Multilingual-Multimodal-NLP/cyberfactory https://github.com/CSJianYang/CyberFactory