Guidelines for Evaluating Large Language Models in Cybersecurity Tasks

Large language models (LLMs) demonstrate remarkable abilities to ingest, synthesize, and summarize knowledge, yet they show clear limitations when performing concrete tasks. In cybersecurity, LLMs can help experts prevent and stop attacks, but adversaries can also exploit generative AI, as seen with threats like WormGPT and FraudGPT.

Can LLMs such as GPT‑4 write novel malicious software?

Will LLMs become a core component of large‑scale cyber attacks?

Can we trust LLMs to provide reliable information to security professionals?

The answers depend on the evaluation methods used, and current techniques are not comprehensive. Researchers from SEI CERT, in collaboration with OpenAI, have developed improved methods and distilled fourteen recommendations to help assess LLMs' cybersecurity capabilities.

Challenges of Using LLMs in Cybersecurity Tasks

Real‑world security tasks are complex and dynamic, involving an ongoing interplay between attackers and defenders. Attackers may change tactics based on defender actions, and defenders must decide whether to observe, gather intelligence, or immediately block intrusions. Using LLMs raises questions such as whether they can suggest or take actions faster and more effectively, and whether they might cause unintended or destructive outcomes.

Existing evaluations often rely on simple fact‑retrieval benchmarks, which do not reflect the nuanced decision‑making required in actual security operations. Consequently, decision‑makers lack the information needed to gauge both the opportunities and risks of deploying LLMs in sensitive environments.

Evaluation Recommendations

To accurately assess the risks and suitability of LLMs for security tasks, evaluators should consider the following guidance:

Define the real tasks you want to capture. Start by describing how humans perform the task, breaking it down into concrete steps.

Use existing datasets cautiously. Many security evaluations rely on public datasets, which can limit the scope and quality of the assessment.

Specify the intended use. Clarify whether you are interested in autonomous behavior or human‑AI collaboration, as this shapes the evaluation design.

Represent Tasks Appropriately

Security tasks are often too nuanced for simple multiple‑choice questions. Instead, design queries that reflect the task’s complexity without artificially constraining it. Guidelines include:

Determine an appropriate scope; avoid overly narrow sub‑tasks that do not map to real‑world objectives.

Build supporting infrastructure; realistic evaluations may require extensive setup to enable interaction between the LLM and a test environment.

Incorporate realistic human capabilities where relevant, ensuring the evaluation mirrors actual human endurance and adaptability.

Avoid unrealistic human assumptions; professional certification settings may overlook real‑world constraints.

Make Your Evaluation More Robust

When designing assessments, follow these principles:

Use pre‑registration to define scoring in advance.

Apply realistic perturbations to inputs (e.g., rephrasing, reordering) to test model stability.

Guard against training‑data contamination, as LLMs may have seen vulnerability disclosures, CVE listings, or code snippets that could bias results.

Set an Appropriate Framework for Results

Interpretation should avoid overgeneralization and consider best‑case and worst‑case performance, model‑selection bias, and whether the focus is on risk assessment or capability profiling.

Summary and Outlook

Without careful risk management, AI and LLMs can be either valuable assets for security professionals or powerful tools for malicious actors. Developing realistic, competitive‑scenario evaluations will provide clearer insight into LLM capabilities and inform policy decisions.

The article concludes by promoting the upcoming R&D Efficiency & Innovation Conference, which will cover topics such as AIGC, BizDevOps, AI‑human collaboration, B‑to‑B product management, platform engineering, and more. Attendees are invited to join interactive sessions, workshops, and to consult the provided conference guide for scheduling details.