Refusal ≠ Safety: ReasoningShield Exposes Chain-of-Thought Security Blind Spots

As large reasoning models adopt "think-then-answer" patterns, safety auditing must examine the entire reasoning trace, not just the final answer. A model may refuse a harmful request in its final output while having already disclosed actionable dangerous information during intermediate reasoning steps. Traditional moderation tools that only inspect final answers miss this risk.

Problem: Reasoning Traces Contain Hidden Harm

Reasoning traces include hypotheses, self-corrections, and alternating safety statements with harmful details. A model might first discuss a dangerous approach, then realize the issue and refuse. The conclusion is compliant, but the intermediate steps may have already leaked usable harmful information. This creates two challenges for existing moderators: (1) risk is scattered across long intermediate steps and can be masked by a safe ending; (2) "discussing risk" and "providing harmful guidance" use similar vocabulary, causing confusion.

Preliminary experiments show that existing moderation tools suffer a maximum F1 drop of 35.2 percentage points when moved from final-answer moderation to chain-of-thought moderation, proving that chain-of-thought safety auditing cannot simply reuse final-answer evaluation methods. Effective auditing must combine user intent, reasoning context, and actionability of information to judge whether risk has already materialized in intermediate steps.

First Chain-of-Thought Safety Benchmark: 9.2K Samples, 10 Risk Categories, 3-Level Labels

The team merged academic safety benchmarks with industry content policies to build a risk taxonomy covering 10 top-level categories (violence, hate & discrimination, deception & misinformation, rights violation, cybersecurity, etc.) and 42 subcategories . Reasoning traces are labeled at three levels: Safe (0), Potentially Harmful (0.5), Harmful (1) . The "Potentially Harmful" level captures borderline cases where information is disclosed but not yet actionable guidance, preserving risk granularity and allowing users to adjust strictness.

Based on this taxonomy, they constructed ReasoningShield-Train (7,000 samples) and ReasoningShield-Test (2,200 samples) . The training set covers 4 open-source reasoning models; the test set extends to 8 reasoning models, including 600 in-domain and 1,600 out-of-distribution samples to test generalization across sources and reasoning paradigms. Label distribution in train: ~35.2% Harmful, 19.8% Potentially Harmful, 45.0% Safe; test: ~28.6% Harmful, 15.2% Potentially Harmful, 56.3% Safe. Additionally, 1,064 harmless Alpaca instructions were added to prevent over-fitting to risky samples.

Annotation used a human-in-the-loop pipeline: multi-model voting, expert review, and quality audit. The test set was independently annotated by three AI safety researchers. The benchmark provides not only final labels but also structured step-by-step risk analyses for training.

Model Training: SFT + DPO Two-Stage Approach

Models are based on Llama-3.2-1B/3B . Stage 1 (SFT): supervised fine-tuning on agreed-upon samples teaches the model a structured moderation workflow — user intent identification → step-by-step risk analysis → comprehensive judgment → safety label . Stage 2 (DPO): on difficult samples with annotation disagreement, preference pairs of positive/negative analyses and judgments are built; DPO increases preference for high-quality moderation, sharpening boundary detection for potentially harmful content. Ablation shows the two-stage pipeline outperforms SFT alone on both chain-of-thought and answer moderation.

Experimental Results: Specialist Models Excel on Reasoning Traces

On the ReasoningShield-Test open-source model trace subset , ReasoningShield-3B achieves 91.8% F1 (SOTA) . In contrast, LlamaGuard-4-12B scores 56.2% ; general-purpose models with the same moderation prompt: GPT-4o 87.8% , Qwen2.5-72B 89.0% . Under controlled conditions (same Llama-3.2 base, similar SFT+DPO recipe), ReasoningShield consistently outperforms the safety moderator GuardReasoner across multiple datasets.

Robust Generalization: Cross-Paradigm, Cross-Task, Out-of-Distribution, Adversarial

Cross reasoning paradigm: Trained on 4 open-source reasoning models, tested on 4 commercial models (Claude, Gemini, Grok, Doubao) with different trace lengths, styles, and organization. ReasoningShield-3B averages 81.4% F1 vs. LlamaGuard-4-12B's 33.7%.

Cross-task transfer: Applied directly to WildGuard-Test and BeaverTails-Test (answer moderation) without retraining. Overall F1 85.0% ; on WildGuard-Test, 86.4% F1 vs. official WildGuard's 75.7% (+10.7 pp).

Out-of-distribution: Evaluated on unseen BeaverTails and Jailbreak query sources. On corresponding open-source trace subsets, ReasoningShield-3B reaches 90.4% and 92.3% F1 .

Adversarial robustness: 100 risk traces perturbed with 5 obfuscation strategies (sensitive word replacement, syntax restructuring, noise insertion, partial translation, style rewrite) → 500 adversarial samples . ReasoningShield-3B maintains 93.2% F1 vs. LlamaGuard-4-12B's 50.0%. Extending traces to ~10K tokens to dilute local risk, ReasoningShield-3B still holds 89.7% F1 vs. LlamaGuard-4-12B's 28.1%.

Explainable Moderation: Pinpointing Risk Location

ReasoningShield takes the user question and full reasoning trace as input, performing intent recognition, step-by-step risk analysis, comprehensive judgment, and safety grading. On 200 stratified traces, human verification shows 89.2% evidence hit rate on 120 risky traces (correctly locating supporting evidence), and zero false risk attributions on 80 safe traces . In a privacy case where reasoning masqueraded as educational discussion but gradually probed for private information sources, ReasoningShield pinpointed the risk to step 3, enabling targeted human review.

Conclusion and Future Work

Risk in reasoning is dynamic, unfolding and propagating through context. Effective moderation must see both local steps and global intent, provide conclusions with evidence. Remaining challenges: risks relying on external knowledge (cultural metaphors, implicit bias) may still be missed; future work should explore retrieval augmentation, continuous updating, and extension to multilingual, multimodal reasoning, and cross-domain applications. As reasoning models grow more capable, safety supervision must shift from result analysis to process analysis — trustworthy reasoning should not only produce safe answers but also leave potential risks traceable and auditable.

Paper: https://arxiv.org/abs/2505.17244

Project page: https://cosmosyi.github.io/ReasoningShield/

GitHub: https://github.com/CosmosYi/ReasoningShield

Models: https://huggingface.co/ReasoningShield/ReasoningShield-1B, https://huggingface.co/ReasoningShield/ReasoningShield-3B

Dataset: https://huggingface.co/datasets/ReasoningShield/ReasoningShield-Dataset