让大模型先判断安全再推理，提升安全性同时不损推理能力
Towards Safer Large Reasoning Models by Promoting Safety Decision-Making before Chain-of-Thought Generation
- 先让模型做安全判断，再生成思维链，避免安全漏洞
- 实验显示安全性显著提升，推理能力基本不变
- 适合关注AI安全的开发者和研究者
大型推理模型（LRMs）通过思维链（CoT）实现优异表现，但近期研究发现，启用CoT后其安全能力显著下降。本文发现，这种安全退化仅在启用CoT时出现，禁用时则不明显。为此，我们提出一种新方法：在生成思维链前，促使模型优先做出安全决策。具体而言，使用基于Bert的分类器从安全模型（如禁用CoT的LRM）中提取安全判断信号，并作为辅助监督信号融入对LRM的安全对齐过程。该机制使安全梯度可反向传播至模型隐状态，有效强化模型在生成思维链前的安全判断能力。大量实验证明，该方法显著提升了LRM的安全性，同时保持了其通用推理性能。
原文摘要 · Abstract (English)
Large reasoning models (LRMs) achieved remarkable performance via chain-of-thought (CoT), but recent studies showed that such enhanced reasoning capabilities are at the expense of significantly degraded safety capabilities. In this paper, we reveal that LRMs' safety degradation occurs only after CoT is enabled, and this degradation is not observed when CoT is disabled. This observation motivates us to consider encouraging LRMs to make safety decisions before CoT generation. To this end, we propose a novel safety alignment method that promotes the safety decision-making of LRMs before starting CoT generation. Specifically, we first utilize a Bert-based classifier to extract safety decision signals from a safe model (e.g., a CoT-disabled LRM) and then integrate these signals into LRMs' safety alignment as auxiliary supervision. In this way, the safety gradients can be backpropagated to the LRMs' latent representations, effectively strengthening the LRMs' safety decision-making abilities against CoT generation. Extensive experiments demonstrate that our method substantially improves the safety capabilities of LRMs while effectively maintaining LRMs' general reasoning performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。