通过筛选高梯度样本，实现大模型持续学习中的安全对齐
Continual Safety Alignment via Gradient-Based Sample Selection
- 用梯度大小筛选训练样本，避免有害微调
- 高梯度样本导致安全退化，中梯度样本更优
- 无需额外安全数据，适合持续学习场景
大语言模型在持续适应新任务时需保持安全对齐。然而，即使使用无害数据微调，也常导致安全行为退化，包括拒绝有害请求、真实性及常识推理能力下降。我们从数据视角分析哪些样本引发对齐漂移：实验表明，高梯度样本造成更大安全损失，并使模型趋向预训练分布；而中梯度样本可在最小对齐损失下实现任务学习。为此提出基于梯度的样本选择方法，在微调中过滤高梯度样本。在多个模型家族和连续领域任务上，该方法显著提升对齐保留能力，同时保持良好任务性能，无需定制安全数据或架构修改。方法在不同选择比例、任务顺序和攻击基准下均表现稳健。
原文摘要 · Abstract (English)
Large language models require continuous adaptation to new tasks while preserving safety alignment. However, fine-tuning on even benign data often compromises safety behaviors, including refusal of harmful requests, truthfulness, and commonsense reasoning. We investigate which training samples cause alignment drift through a data-centric lens. Our empirical analysis shows samples contribute unequally: high-gradient samples cause greater safety degradation and drive models toward pretrained distributions, while moderate-gradient samples enable task learning with minimal alignment loss. We propose gradient-based sample selection that filters high-gradient samples during fine-tuning. Across multiple model families on continual domain tasks, our method substantially improves alignment preservation while maintaining competitive task performance, without requiring curated safe data or architectural modifications. Our method is robust across selection ratios, task orderings, and diverse attack benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。