通过识别高冲突头,精准微调提升模型安全与能力的平衡。
Safety-Utility Conflicts Are Not Global: Surgical Alignment via Head-Level Diagnosis
- 按注意力头分级诊断冲突,只更新低冲突部分。
- 跳过少数高冲突头,通用能力下降减少70%以上。
- 适合追求高效、可解释对齐的模型优化场景。
大语言模型的安全对齐存在多目标优化冲突,常伴随通用能力下降。现有方法依赖全局梯度几何,却忽视了Transformer中不同注意力头的功能敏感性和冲突程度差异。此类全局策略对所有参数施加统一更新规则,常导致对敏感头的无效更新。为此,我们提出冲突感知稀疏微调(CAST),通过融合优化冲突与功能敏感性构建预对齐冲突图,指导参数选择性更新。实验表明,对齐冲突并非均匀分布;通用能力下降主要源于少数“高冲突”头的更新。仅跳过这些头进行训练,即可显著降低能力损失,且不损害安全性,实现可解释、高效的安全部署。
原文摘要 · Abstract (English)
Safety alignment in Large Language Models (LLMs) inherently presents a multi-objective optimization conflict, often accompanied by an unintended degradation of general capabilities. Existing mitigation strategies typically rely on global gradient geometry to resolve these conflicts, yet they overlook Modular Heterogeneity within Transformers, specifically that the functional sensitivity and degree of conflict vary substantially across different attention heads. Such global approaches impose uniform update rules across all parameters, often resulting in suboptimal trade-offs by indiscriminately updating utility sensitive heads that exhibit intense gradient conflicts. To address this limitation, we propose Conflict-Aware Sparse Tuning (CAST), a framework that integrates head-level diagnosis with sparse fine-tuning. CAST first constructs a pre-alignment conflict map by synthesizing Optimization Conflict and Functional Sensitivity, which then guides the selective update of parameters. Experiments reveal that alignment conflicts in LLMs are not uniformly distributed. We find that the drop in general capabilities mainly comes from updating a small group of ``high-conflict'' heads. By simply skipping these heads during training, we significantly reduce this loss without compromising safety, offering an interpretable and parameter-efficient approach to improving the safety-utility trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。