Cybersecurity LLM Data Acquisition: 7 Sources with Reproducible Code

Introduction: Why Cybersecurity and Why Small Models

The article begins by explaining the rationale for choosing cybersecurity as the vertical domain for an end-to-end LLM training demonstration. Two key reasons are given: (1) cybersecurity is one of the fastest areas where LLMs show tangible impact, as demonstrated by Claude Mythos in reverse engineering and vulnerability discovery; (2) small-parameter models can be highly effective—the paper VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detection (GitHub: https://github.com/ucsb-mlsec/VulnLLM-R) shows a 7B model achieving strong vulnerability detection results through careful data curation, proving that a "precision cultivation" approach works without massive parameters.

The first step in training any vertical model is data. This article systematically catalogs the data sources needed for pre-training a cybersecurity LLM, giving concrete acquisition channels and runnable code examples for each category.

1. Data Source Catalog: Where to Find the "Textbooks" for a Cybersecurity LLM

Cybersecurity data is characterized as scattered, heterogeneous, and time-sensitive. The author emphasizes that deep understanding of the domain's knowledge system and data distribution is the real threshold for data engineering.

1.1 Electronic Books: Systematic Knowledge Foundation

High-quality books provide structured, coherent, and deep coverage that fragmented blogs cannot match. Their clear chapter hierarchy becomes a natural knowledge graph after conversion to Markdown, aiding downstream chunking and distillation. The author cites Anthropic's multi-million-dollar purchase of physical books for training as evidence of book corpora's value.

Beyond security-specific titles, foundational textbooks on computer networks, operating systems, and computer architecture are equally essential because security knowledge is tightly interwoven with these basics.

Recommended sources fall into three groups:

Security professional books: Wu Hanqing's White Hat Talks Web Security (Chinese classic); The Web Application Hacker's Handbook (international "bible" for web penetration).

Network fundamentals: Xie Xiren's Computer Networks (standard Chinese textbook); Tanenbaum's Computer Networks and Stevens' TCP/IP Illustrated, Vol. 1 (international classics, the latter uses real packet captures).

Computer fundamentals: CSAPP ( Computer Systems: A Programmer's Perspective ); Silberschatz's Operating System Concepts ("Dinosaur Book").

Selection criteria: prefer authoritative publishers (Posts & Telecom, Machinery Industry, Electronic Industry, O'Reilly), authors with real engineering/academic track records, multiple editions, high Douban ratings or university adoption, and clear structure/terminology/rigorous figures after trial reading.

Special reminder: Books are copyrighted. Personal practice is usually fine, but commercial model use requires verifying that the license explicitly permits training use.

1.2 Academic Papers & Top Conference Proceedings: Cutting-Edge Foundations

Papers provide three irreplaceable values: (1) principled foundations—teaching the model why attacks/defenses work; (2) peer-reviewed credibility with experimental data, reducing hallucinations; (3) frontier timeliness—arXiv preprints and top conferences cover directions like LLM security, AI-driven vulnerability mining, and jailbreak attacks years before blogs.

Two primary channels:

arXiv cs.CR (Cryptography and Security): main preprint venue, fast updates, free search API for batch collection.

Four top security conferences: IEEE S&P, USENIX Security, ACM CCS, NDSS.

Example: fetching arXiv papers related to "log4j" in cs.CR using Python:

# arxiv_search.py: search arXiv cs.CR for security papers
import urllib.request
import urllib.parse
import xml.etree.ElementTree as ET

query = urllib.parse.quote("cat:cs.CR AND all:log4j")
url = (f"http://export.arxiv.org/api/query?search_query={query}"
       f"&start=0&max_results=2")
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
with urllib.request.urlopen(req, timeout=30) as resp:
    root = ET.fromstring(resp.read())

ns = {"a": "http://www.w3.org/2005/Atom"}
for entry in root.findall("a:entry", ns):
    raw_id = entry.find("a:id", ns).text.strip()
    arxiv_id = raw_id.split("/abs/")[-1]
    title = entry.find("a:title", ns).text.strip().replace("
", " ")
    summary = entry.find("a:summary", ns).text.strip().replace("
", " ")
    print("Paper ID:", arxiv_id)
    print("Title:", title)
    print("Abstract:", summary[:80], "...")
    print("PDF download: https://arxiv.org/pdf/" + arxiv_id)
    print("---")

1.3 NIST, MITRE Standards & Frameworks: Authoritative Normative Knowledge

Standards teach the model "how to do things properly." Two most authoritative sources:

NIST SP 800 series: covers risk management, incident response, cryptographic applications, etc. Example: SP 800-61 Computer Security Incident Handling Guide (free PDF at https://csrc.nist.gov/publications/sp800).

MITRE ATT&CK: a structured knowledge graph organizing attack techniques by tactics, techniques, sub-techniques with real cases and mitigations. Official STIX-format JSON full download at https://github.com/mitre-attack/attack-stix-data. Each technique is a complete structured object.

Example MITRE ATT&CK technique JSON:

{
  "type": "attack-pattern",
  "name": "Phishing: Spearphishing Attachment",
  "description": "Adversaries may send spearphishing emails with a malicious attachment...",
  "kill_chain_phases": [{"phase_name": "initial-access"}]
}

In practice, concatenate name, description, and mitigations into natural-language paragraphs for high-quality corpus.

1.4 CVE, CWE Vulnerability Databases: Core Competitive Advantage

Vulnerability knowledge is a core differentiator for cybersecurity LLMs. Users ask: "Which CVE does this code trigger?" "How to fix this vulnerability?" The model must answer accurately.

CVE Official API ( https://www.cve.org/): free REST API, no auth, query by ID. Example script for CVE-2021-44228 (Log4Shell):

# cve_search.py: query single CVE record via official API
import json
import urllib.request

CVE_ID = "CVE-2021-44228"
url = f"https://cveawg.mitre.org/api/cve/{CVE_ID}"
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
with urllib.request.urlopen(req, timeout=30) as resp:
    data = json.load(resp)

cna = data["containers"]["cna"]
desc = next((d["value"] for d in cna["descriptions"] if d["lang"].startswith("en")), "")
affected = cna["affected"][0]

print("ID:", data["cveMetadata"]["cveId"])
print("Title:", cna.get("title", ""))
print("Vendor/Product:", affected["vendor"], "/", affected["product"])
print("Description:", desc[:100], "...")

Structured fields can be turned into natural language via an LLM, e.g.: "CVE-2021-44228 is a JNDI injection vulnerability in Apache Log4j2, remotely exploitable without authentication; upgrade to 2.17.1 or later."

For bulk collection, clone the official mirror CVEProject/cvelistV5 ( https://github.com/CVEProject/cvelistV5.git) and filter by year, vendor, etc.

CWE complements CVE: CVE records "which product has which vulnerability," CWE classifies "what type of weakness" (buffer overflow, SQL injection, privilege escalation). Each CWE includes name, detailed description, typical code examples, and mitigation advice—key for teaching root-cause and code-level defense. Example parser:

# cwe_parse.py: download and parse CWE official full data
import urllib.request
import zipfile
import io
import xml.etree.ElementTree as ET

url = "https://cwe.mitre.org/data/xml/cwec_latest.xml.zip"
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
with urllib.request.urlopen(req, timeout=120) as resp:
    data = resp.read()

with zipfile.ZipFile(io.BytesIO(data)) as zf:
    name = [n for n in zf.namelist() if n.endswith(".xml")][0]
    root = ET.fromstring(zf.read(name))

ns = {"c": "http://cwe.mitre.org/cwe-7"}
weaknesses = root.find("c:Weaknesses", ns)
for w in list(weaknesses)[:3]:
    desc = w.find("c:Description", ns).text.strip().replace("
", " ")
    print(f"CWE-{w.get('ID')} | {w.get('Name')} | {desc[:60]}...")
print("Total weaknesses:", len(weaknesses.findall("c:Weakness", ns)))

Nearly a thousand CWE definitions, each a complete "definition + code example + fix" structure, become systematic vulnerability-root-cause corpus after templating. Exploit-DB (public exploit code repository) further helps the model understand actual exploitation for better defensive advice.

1.5 Technical Communities & Vendor Blogs: Live Battle Experience

Papers and standards give "knowledge," but security is practice-driven—a real incident retrospective can outweigh ten papers. This corpus emphasizes practicality, timeliness, and case detail.

Chinese communities: FreeBuf, SecPulse, XZ (Xianzhi) — abundant vulnerability analyses, penetration tips, incident retrospectives (e.g., FreeBuf's Log4Shell technical analyses).

Vendor threat reports: QiAnXin, 360, Weibu Online — periodic APT group analyses, annual security reports (e.g., QiAnXin's annual Cybersecurity Threat Analysis Report ).

English channels: Krebs on Security, Google Project Zero blog.

Warning: This category has the largest volume and most noise—reposts, ads, clickbait—making it a key focus for the cleaning stage (covered next).

2. Ready-Made Datasets: Standing on the Shoulders of the Community

Beyond raw collection, two high-value shortcuts exist: model distillation to create custom datasets, and direct use of open-source community datasets.

2.1 EasyDataset Distillation: Using Large Models to Manufacture Data

Closed-source and massive open models lead in performance. Via model distillation , large models can "condense" their reasoning into high-quality datasets, letting small models approximate large-model performance on specific tasks. A notable example: Fei-Fei Li's team's paper s1: Simple test-time scaling spent ~$50 to fine-tune Qwen2.5-32B on data partly distilled from Google Gemini 2.0 Flash Thinking, reaching reasoning parity with ChatGPT o1 and DeepSeek R1.

The author's previously introduced EasyDataset tool implements distillation. Its mechanism: build a multi-level label tree covering the task domain, then auto-generate Q&A pairs for each leaf label, ensuring coverage, diversity, and balance.

Demonstration for "penetration testing" scenario:

Create project, set top-level theme: Name project "Web Security Offensive/Defensive: Penetration Testing Practical Guide" (borrowing a known book title as anchor) and paste the book's full table of contents into the description to give the LLM a clear knowledge boundary.

Enter distillation module, configure parameters: Two key settings: (a) label hierarchy depth and labels per level—deeper levels yield finer-grained knowledge units (e.g., 2 levels: "Web Security → SQL Injection"; 3-4 levels: "Web Security → SQL Injection → Error-based Injection → Error Function Exploitation"); (b) questions per label—controls final dataset size. Real-time preview shows estimated total questions.

Launch distillation, track progress: UI shows detailed progress across label-building, question-generation, and answer-generation phases with completion counts.

Optional manual fine-grained adjustment: EasyDataset also supports manual step-by-step label definition with human-in-the-loop at each level, suitable when domain knowledge is well-understood.

2.2 Open-Source Papers & Community Datasets

Hugging Face / ModelScope security datasets: Search "cybersecurity", "安全" for QA pairs, vulnerability detection code, etc. Example: the VulnLLM-R paper's dataset is on Hugging Face (

https://hf-mirror.com/datasets/UCSB-SURFI/Reduced-Distill-DeepSeek/viewer/default/train?row=0

).

CTF competition writeups: Post-competition solutions from major CTF platforms are excellent "offensive/defensive hands-on" corpus; collect separately.

Practical tip: When using Hugging Face datasets, first check label distribution, data volume, and license (commercial use allowed? matches training goal?) to avoid compliance risks later.

3. Summary

This installment kicks off the practical chapter, systematically covering seven core data sources for a cybersecurity LLM: academic papers & top conferences, NIST/MITRE standards, CVE/CWE databases, electronic books, technical communities & vendor reports, open-source datasets, and EasyDataset-distilled custom corpora—each with concrete channels and reproducible collection code.

Data collected, but still raw heterogeneous material: papers are two-column PDFs needing layout parsing; e-books mix text, tables, code blocks; community articles contain duplicates, ads, clickbait. Unprocessed, these noise and format barriers directly degrade training quality.

Next episode "Pre-training Data Processing" will show how to turn these rough materials into high-quality datasets the model can digest. Stay tuned.