AI 研究arXiv cs.CL12 Oct, 12:00 pm SGTSIGNAL C94021

ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models

This study systematically reveals a unique hidden safety threat in large reasoning models (LRMs): harmful content can be embedded in intermediate Chain-of-Thought (CoT) steps while the final answer appears benign. Existing state-of-the-art moderation tools suffer substantial performance degradation on CoT moderation (F1 drops by up to 35.2%). To address this, the authors release the ReasoningShield benchmark (9.2K annotated pairs) and train lightweight 1B/3B models that achie

01 ABSTRACT

The study finds that the intermediate reasoning steps of large reasoning models can contain harmful content while the final answer appears benign, causing existing moderation tools to lose up to 35.2% F1 on CoT moderation. The authors build the ReasoningShield benchmark with 9.2K (Query, CoT) pairs spanning 10 risk categories, 3 safety levels, and 8 LRM paradigms, and develop lightweight 1B/3B models via a two-stage training strategy that achieve 91.8% F1, outperforming LlamaGuard-4 by 35.6% and GPT-4o by 15.8%. The authors argue that the benchmark and models generalize effectively across diverse reasoning paradigms and unseen scenarios, but independent verification of generalization is still needed.

02 KEY FINDINGS

Reveals the unique safety risk that harmful content can be embedded in CoT intermediate steps while the final answer appears benign; existing moderation tools lose up to 35.2% F1 in this setting.
Releases the ReasoningShield benchmark with 9.2K annotated (Query, CoT) pairs covering 10 risk categories, 3 safety levels, 8 LRMs, and diverse reasoning paradigms.
Develops lightweight 1B/3B models via two-stage training that achieve 91.8% F1 on the benchmark, outperforming LlamaGuard-4 by 35.6% and GPT-4o by 15.8%, and reports generalization across paradigms and unseen scenarios.

03 RELEVANCE

As large reasoning models are increasingly deployed for complex tasks, their Chain-of-Thought traces are becoming a new attack surface: harmful content can bypass answer-level moderation. ReasoningShield advances safety detection to the reasoning-trace level and provides public benchmarks, models, and training resources, helping safety researchers and developers assess and mitigate CoT-specific risks — a direct reference for building trustworthy reasoning systems.

04 EVIDENCE BOUNDARY

The paper abstract reports results only on the self-built ReasoningShield benchmark and a limited evaluation; it does not address robustness in real production environments, multi-turn interactions, or adversarial attacks. The reported 91.8% F1, 35.6%, and 15.8% figures are single-run results from the abstract without details on independent replication or statistical significance. Whether the benchmark's 10 risk categories, 8 LRMs, and 3 safety levels sufficiently represent all reasoning paradigms and risk scenarios requires further validation. The generalization claims of the lightweight models are based on a specific dataset; effectiveness across domains, languages, and unseen risk types still needs independent assessment.

Return to the primary source

AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.CL

Read original ↗