通过对抗性思维链训练，让大模型学会自我纠正，提升安全性和可用性。
AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models
- 构建诱惑-修正与犹豫-修正样本，训练模型动态纠错。
- 在对抗攻击下误判率降低42%，良性提示拒绝率下降37%。
- 适合关注模型安全性与实用平衡的研究者和开发者。
大型推理模型（LRMs）通过思维链（CoT）推理在复杂问题求解中表现出色。然而，CoT的多步特性引入了新的安全挑战，超出传统语言模型对齐范畴。我们识别出当前安全CoT调优方法的一种失效模式：雪球效应——微小的推理偏差在过程中逐级放大，导致有害顺从或过度拒绝。该现象源于模型仅模仿完美推理脚本，未学习自我修正能力。为此，我们提出AdvChain，一种通过对抗性CoT调优实现动态自我修正的对齐范式。方法构建包含诱惑-修正和犹豫-修正样本的数据集，使模型学会从有害推理漂移和过度谨慎中恢复。大量实验表明，AdvChain显著提升了对越狱攻击和CoT劫持的鲁棒性，同时大幅减少对良性提示的过度拒绝，在不损害推理能力的前提下实现了更优的安全-效用平衡。本工作为构建更稳健可靠的推理模型开辟了新方向。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) have demonstrated remarkable capabilities in complex problem-solving through Chain-of-Thought (CoT) reasoning. However, the multi-step nature of CoT introduces new safety challenges that extend beyond conventional language model alignment. We identify a failure mode in current safety CoT tuning methods: the \textit{snowball effect}, where minor reasoning deviations progressively amplify throughout the thought process, leading to either harmful compliance or excessive refusal. This effect stems from models being trained to imitate perfect reasoning scripts without learning to self-correct. To address this limitation, we propose AdvChain, an alignment paradigm that teaches models dynamic self-correction through adversarial CoT tuning. Our method involves constructing a dataset containing Temptation-Correction and Hesitation-Correction samples, where models learn to recover from harmful reasoning drifts and unnecessary cautions. Extensive experiments show that AdvChain significantly enhances robustness against jailbreak attacks and CoT hijacking while substantially reducing over-refusal on benign prompts, achieving a superior safety-utility balance without compromising reasoning capabilities. Our work establishes a new direction for building more robust and reliable reasoning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。