ReasoningShield: Why Final Answer Safety Isn't Enough for Chain-of-Thought Models

As large reasoning models adopt chain-of-thought (CoT) generation, a critical safety gap emerges: a model may refuse a harmful request in its final answer while having already disclosed actionable dangerous information during intermediate reasoning steps. Traditional safety moderators that only inspect final answers miss this exposure.

Problem: Reasoning Traces Leak Harm Before Refusal

CoT traces contain hypotheses, self-corrections, and alternating safety declarations with harmful details. A model can discuss a dangerous approach, realize the risk, and then refuse—but the intermediate steps may already have revealed exploitable information. Existing moderators face two challenges: (1) risk signals are scattered across long traces and masked by a safe conclusion; (2) "discussing risk" and "providing harmful guidance" use similar vocabulary, causing confusion. Pilot experiments show that when existing answer-level moderators are applied to CoT traces, F1 drops by up to 35.2 percentage points .

ReasoningShield Benchmark: 9.2K Samples, 10 Risk Categories, 3-Level Labels

The team from Shanghai AI Laboratory and Fudan University merged academic safety benchmarks with industry content policies to create a taxonomy of 10 top-level risk categories (violence, hate & discrimination, deception & misinformation, rights violation, cybersecurity, etc.) spanning 42 subcategories . Each reasoning trace receives a three-level label: Safe (0) , Potentially Harmful (0.5) —discloses abusable information without direct action guidance—and Harmful (1) . This granularity preserves risk nuance and lets downstream users adjust strictness.

Two datasets were built:

ReasoningShield-Train : 7,000 traces from 4 open-source reasoning models; label distribution ~35.2% Harmful / 19.8% Potentially Harmful / 45.0% Safe.

ReasoningShield-Test : 2,200 traces from 8 reasoning models (600 in-domain, 1,600 out-of-distribution); distribution ~28.6% / 15.2% / 56.3%.

An additional 1,064 benign Alpaca instructions were included to prevent over-triggering on normal reasoning. Annotation used a human–machine pipeline: multi-model voting, expert review, and quality audit for training data; three AI safety researchers independently labeled the test set. The benchmark also provides structured step-by-step risk analyses for training.

Model Training: Two-Stage SFT + DPO on Llama-3.2

Using Llama-3.2-1B/3B as base models, the team employed:

Supervised Fine-Tuning (SFT) on high-agreement samples to teach a structured moderation flow: user intent identification → step-wise risk analysis → holistic judgment → safety label .

Direct Preference Optimization (DPO) on low-agreement "hard" samples, constructing chosen/rejected analysis-judgment pairs to sharpen boundary discrimination, especially for Potentially Harmful cases.

Ablation confirms the two-stage pipeline outperforms SFT alone on both CoT and answer moderation.

Experimental Results: SOTA on Reasoning Traces, Strong Generalization

In-Domain (Open-Source Model Traces)

On the ReasoningShield-Test open-source subset, ReasoningShield-3B reaches 91.8% F1 , establishing a new state of the art. For comparison:

LlamaGuard-4-12B: 56.2%

GPT-4o (with same moderation prompt): 87.8%

Qwen2.5-72B: 89.0%

Under a controlled comparison (same Llama-3.2 base, similar SFT+DPO recipe), ReasoningShield also beats GuardReasoner across multiple datasets.

Cross-Paradigm Generalization (Commercial Models)

Training used only 4 open-source models; test adds traces from Claude, Gemini, Grok, and Doubao . ReasoningShield-3B averages 81.4% F1 vs. LlamaGuard-4-12B's 33.7%, showing robustness to different trace lengths, phrasing styles, and reasoning structures.

Cross-Task Generalization (Answer-Level Moderation)

Without retraining, the same model applied to WildGuard-Test and BeaverTails-Test achieves 85.0% overall F1 . On WildGuard-Test specifically, it scores 86.4% F1 , a 10.7-point gain over the official WildGuard model (75.7%).

Out-of-Distribution Generalization

Evaluated on unseen BeaverTails and Jailbreak query sources (open-source model traces): 90.4% and 92.3% F1 respectively.

Adversarial Robustness

Five perturbation types (sensitive-word replacement, syntax restructuring, noise insertion, partial translation, style rewrite) produced 500 adversarial traces preserving original harm semantics. ReasoningShield-3B maintains 93.2% F1 ; LlamaGuard-4-12B drops to 50.0%. Extending traces to ~ 10K tokens to dilute local risk yields 89.7% F1 for ReasoningShield-3B vs. 28.1% for LlamaGuard-4-12B.

Explainable Moderation: Pinpointing Risk to Specific Steps

ReasoningShield takes the user question and full reasoning trace as input, then outputs: intent recognition, step-wise risk analysis, holistic judgment, and safety level. On 200 stratified traces, human verification shows:

89.2% evidence hit rate on 120 risky traces (model correctly locates the step supporting the risk verdict).

Zero false risk attributions on 80 safe traces.

In a privacy case where reasoning masqueraded as educational discussion but gradually probed for private data sources, ReasoningShield pinpointed the risk to step 3 , enabling targeted human review.

Limitations & Future Work

Failure analysis reveals missed detections on cultural metaphors and implicit biases that rely on external knowledge. The authors suggest exploring retrieval-augmented moderation, continuous update mechanisms, and extending evaluation to multilingual, multimodal reasoning, and cross-domain applications .

Resources: Paper (arXiv:2505.17244), Project page (cosmosyi.github.io/ReasoningShield), GitHub (github.com/CosmosYi/ReasoningShield), HuggingFace models (ReasoningShield/ReasoningShield-1B, -3B) and dataset (ReasoningShield/ReasoningShield-Dataset).