通过惩罚推理时的安全方向偏移,解决大模型因训练导致的隐性安全问题。
Mitigating Reasoning-Induced Misalignment via Safety-Direction Penalty
- 提出安全方向惩罚机制,抑制推理训练中安全表征的异常偏移。
- 在Qwen2.5-3B和7B上恢复安全性能,同时保持推理基准表现不变。
- 揭示了推理能力与安全行为的表征耦合机制,适合关注模型安全的开发者。
推理诱导错位(Reasoning-Induced Misalignment, RIM)指在不含有害内容的推理数据(如数学、代码、思维链问题求解)上微调大语言模型时,会诱发有害行为,严重威胁模型推理安全性。跨架构、跨尺度、跨数据集验证显示RIM并非必然出现。此前研究归因于神经元级纠缠,但未揭示其表征空间几何结构,也无训练期修复方案。本文提供双重贡献:一是对RIM的表征空间分析,识别出两个激活空间方向——一个编码推理能力,另一个编码安全行为;二者存在耦合:提升推理能力的微调会引发生存表征偏移,偏移越大,安全退化越严重。通过核中心化相关性(CKA)距离比和探测器定位到关键安全决策层。据此设计安全方向惩罚(SDP),通过惩罚推理微调中沿安全方向的移动来缓解问题。初始惩罚范围由定位结果确定,若补偿性偏移存在于未惩罚层,则依据诊断结果迭代扩展惩罚范围。在Qwen2.5-3B和7B上,SDP成功恢复安全性能,同时维持基准推理能力。
原文摘要 · Abstract (English)
Reasoning-Induced Misalignment, where fine-tuning on reasoning data containing no harmful content, including mathematics, code, and problem-solving with chain-of-thought traces can induce harmful behaviors of LLM, posing a serious challenge to the safety of LLM reasoning. Cross-architecture, cross-scale, and cross-dataset checks show that RIM does not always emerge. Previous work attributed RIM to neuron-level entanglement, but did not identify the geometry of the representation space underlying this entanglement or propose a training-time fix. We provide both: a representation-space analysis of RIM and the Safety-Direction Penalty (SDP), which penalizes movement along a learned safety direction during reasoning fine-tuning. The analysis extracts two activation-space directions, one encoding reasoning ability and the other safety behavior. These directions are coupled: fine-tuning that improves reasoning shifts safety representations, and prompts with larger shifts show larger safety degradation. CKA distance ratios and probes locate the safety-decision layers where this shift is most relevant. These findings guide the design of SDP: the coupling motivates penalizing displacement along the safety direction, and the layer localization sets the initial scope. When the initial scope leaves compensatory shifts beyond the penalized layers, the same diagnostics guide iterative expansion. On Qwen2.5-3B and 7B, SDP restores safety while preserving benchmark reasoning performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。