微调大模型时用方向约束保住安全，防止有害行为反弹
AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basin
- 微调时只允许沿安全方向更新参数，禁止横向扰动
- 有害行为减少7.60%，任务性能提升3.44%
- 适合注重安全的模型微调场景，如对话系统
微调大语言模型虽能提升性能，却易引入严重安全漏洞：极少量有害数据即可破坏安全机制。我们发现，与对齐方向正交的参数扰动会迅速瓦解模型安全性，而沿对齐方向的更新则能有效保持安全，揭示出参数空间存在‘狭窄安全盆地’。为此，我们提出AsFT（微调中锚定安全），通过显式约束更新方向，惩罚与对齐方向正交的更新，将模型稳定在安全盆地内。在多个数据集和模型上的实验表明，AsFT可使有害行为减少最多7.60%，任务性能提升3.44%，且在多任务上持续优于现有方法。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) improves performance but introduces critical safety vulnerabilities: even minimal harmful data can severely compromise safety measures. We observe that perturbations orthogonal to the alignment direction - defined by weight differences between aligned (safe) and unaligned models - rapidly compromise model safety. In contrast, updates along the alignment direction largely preserve it, revealing the parameter space as a "narrow safety basin". To address this, we propose AsFT (Anchoring Safety in Fine-Tuning) to maintain safety by explicitly constraining update directions during fine-tuning. By penalizing updates orthogonal to the alignment direction, AsFT effectively constrains the model within the "narrow safety basin," thus preserving its inherent safety. Extensive experiments on multiple datasets and models show that AsFT reduces harmful behaviors by up to 7.60%, improves task performance by 3.44%, and consistently outperforms existing methods across multiple tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。