让大模型学会自我审查，提升安全推理能力
Internalizing Safety Understanding in Large Reasoning Models via Verification
- 训练模型自检输出安全性，基于专家推理路径进行自我批判
- 在多种越狱攻击下仍保持高安全性，显著优于传统对齐方法
- 适合需要高可靠性的AI系统开发者和安全研究者
尽管显式思维链（CoT）提升了大模型的推理能力，却也导致最终答案风险上升。当前对齐范式主要依赖外部强制合规，优化模型识别恶意提示而非评估自身输出的安全性。我们发现，看似对齐的模型缺乏内在安全理解，常无法验证自身回答的安全性，且易受对抗性越狱攻击。为此，提出SInternal框架，通过仅在安全验证任务上训练大模型，使其利用专家推理轨迹批判自身生成的答案。实证表明，学习验证能带来强大的泛化能力，显著增强对域外越狱攻击的鲁棒性。此外，结合强化学习时，SInternal作为初始化优于标准监督微调，表明内化安全理解比模仿安全行为更能构建稳固的对齐基础。代码已开源。
原文摘要 · Abstract (English)
While explicit Chain-of-Thought (CoT) empowers large reasoning models (LRMs), it enables the generation of riskier final answers. Current alignment paradigms primarily rely on externally enforced compliance, optimizing models to detect malicious prompts rather than evaluating the safety of their own outputs. We argue that this approach remains largely behavioral: our empirical analysis reveals that ostensibly aligned models lack intrinsic safety understanding, often failing to verify their own response safety and remaining vulnerable to adversarial jailbreaks. To address this fundamental limitation, we propose Safety Internal (SInternal), a framework that internalizes safety specifications by training LRMs exclusively on safety verification tasks to critique their own generated answers using expert reasoning trajectories. We demonstrate that learning to verify induces a strong generalization for response safety, significantly enhancing robustness against out-of-domain jailbreaks. Furthermore, when combined with reinforcement learning, SInternal serves as a superior initialization compared to standard supervised fine-tuning, suggesting that internalizing safety understanding creates a more robust foundation for alignment than merely mimicking safe behaviors. Our codes are available at https://github.com/AlphaLab-USTC/SInternal
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。