发现推理模型安全拒绝能力突然崩溃,揭示其可修复的机制漏洞。
Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?
- 通过追踪每一步的拒绝意图,发现模型在输出前突然放弃拒绝。
- 仅删除3%异常注意力头,攻击成功率降至10%以下。
- 提出新数据筛选法,用1.7%数据达到与全量训练相当的安全效果。
具备多步推理能力的大规模推理模型(LRMs)虽具强大解题能力,但存在令人担忧的安全漏洞且机制不明。本文从机制可解释性视角研究安全对齐失效原因。通过线性探测追踪各词元位置的拒绝意图,发现一种称为“拒绝悬崖”(refusal cliff)的现象:许多对齐不佳的模型在推理过程中正确识别有害提示并保持强烈拒绝意图,但在生成输出前最后几个词元处拒绝分数骤降。这表明模型并非天生不安全,而是拒绝意图被系统性抑制。因果干预分析识别出少数负面贡献的注意力头,仅剔除其中3%即可使攻击成功率低于10%。基于此机制洞察,提出新方法Cliff-as-a-Judge,通过选取具有最大拒绝悬崖的训练样本,高效修复模型安全对齐。该方法仅需1.7%的原始安全训练数据,即实现与全量训练相当的安全提升,验证了安全对齐中‘少即是多’的效应。
原文摘要 · Abstract (English)
Large reasoning models (LRMs) with multi-step reasoning capabilities have shown remarkable problem-solving abilities, yet they exhibit concerning safety vulnerabilities that remain poorly understood. In this work, we investigate why safety alignment fails in reasoning models through a mechanistic interpretability lens. Using a linear probing approach to trace refusal intentions across token positions, we discover a striking phenomenon termed as \textbf{refusal cliff}: many poorly-aligned reasoning models correctly identify harmful prompts and maintain strong refusal intentions during their thinking process, but experience a sharp drop in refusal scores at the final tokens before output generation. This suggests that these models are not inherently unsafe; rather, their refusal intentions are systematically suppressed. Through causal intervention analysis, we identify a sparse set of attention heads that negatively contribute to refusal behavior. Ablating just 3\% of these heads can reduce attack success rates below 10\%. Building on these mechanistic insights, we propose \textbf{Cliff-as-a-Judge}, a novel data selection method that identifies training examples exhibiting the largest refusal cliff to efficiently repair reasoning models' safety alignment. This approach achieves comparable safety improvements using only 1.7\% of the vanilla safety training data, demonstrating a less-is-more effect in safety alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。