arXiv:2509.06795cs.CL2025-09被引 9

通过约束隐藏状态投影方向,防止大模型在微调中丧失拒绝恶意指令的能力。

Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint

  • 引入投影约束损失,稳定模型内部的拒绝方向
  • 实验显示可显著降低安全风险,同时保持任务性能
  • 适合关注模型安全性的研究人员和开发者

指令微调(IFT)虽能有效提升大语言模型(LLM)能力,但会显著削弱其对恶意指令的拒绝能力。研究发现,模型隐藏状态中的拒绝方向(r-direction)在训练过程中易发生漂移,是导致安全风险的关键原因。本文提出ProCon方法,通过在损失函数中加入投影约束项,限制每个样本隐藏状态在r-direction上的投影幅度。分析表明,适当约束可有效缓解方向漂移。为进一步突破性能瓶颈,引入基于数据分布扩展的热启动策略,强化早期约束信号。在多种数据集、场景和模型上实验均验证:ProCon能显著降低IFT带来的安全风险,且优于现有强基线,同时保持任务性能提升。结果表明,该方法有助于稳定r-direction,为未来可解释性驱动的安全研究奠定基础。

原文摘要 · Abstract (English)

Instruction Fine-Tuning (IFT) has been widely adopted as an effective post-training strategy to enhance various abilities of Large Language Models (LLMs). However, prior studies have shown that IFT can significantly compromise LLMs' safety, particularly their ability to refuse malicious instructions, raising significant concerns. Recent research into the internal mechanisms of LLMs has identified the refusal direction (r-direction) in the hidden states, which plays a pivotal role in governing refusal behavior. Building on this insight, our study reveals that the r-direction tends to drift during training, which we identify as one of the causes of the associated safety risks. To mitigate such drift, our proposed ProCon method introduces a projection-constrained loss term that regularizes the projection magnitude of each training sample's hidden state onto the r-direction. Our initial analysis shows that applying an appropriate constraint can effectively mitigate the refusal direction drift and associated safety risks, but remains limited by overall performance barriers. To overcome this barrier, informed by our observation of early-stage sharp drift and a data-driven perspective, we introduce a warm-up strategy that emphasizes early-stage strong constraints and broaden the data distribution to strengthen constraint signals, leading to an enhanced ProCon method. Experimental results under various datasets, scenarios, and LLMs demonstrate that our method can significantly mitigate safety risks posed by IFT while preserving task performance gains. Even compared with strong baselines, our method consistently delivers superior overall performance. Crucially, our analysis indicates that ProCon can contribute to stabilizing the r-direction during training, while such an interpretability-driven exploration of LLMs' internal mechanisms lays a solid foundation for future safety research.

模型安全微调拒绝行为可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。