用分阶段学习提升大模型安全对齐的鲁棒性，减少有害回应和越狱攻击。
Curriculum Learning for Safety Alignment
- 按难度组织偏好数据，分阶段逐步训练，动态更新参考模型。
- 跨三类模型平均降低16%的域外有害响应，20%的越狱成功率。
- 仅需75%数据达同等安全效果，适合安全对齐研究者使用。
直接偏好优化（DPO）广泛用于大语言模型的安全对齐，但现有研究表明其脆弱且泛化能力差。本文探讨课程学习能否提升基于DPO的安全对齐鲁棒性。提出一种名为分阶段能力（Staged-Competence）的课程框架，通过按难度组织偏好数据、基于能力采样，并在训练中逐步更新参考模型。在三个模型族上平均，该方法将域外有害响应率降低16%，越狱攻击成功率下降20%，同时保持通用能力，过拒率接近零。进一步表明，该方法仅需75%训练数据即可达到基线安全水平，并实现更优的安全与不安全响应分离。该框架与策略优化损失无关，可扩展至其他DPO变体及对齐领域。代码与数据已公开于https://github.com/Sandeep5500/curriculum-learning-for-safety。
原文摘要 · Abstract (English)
Direct Preference Optimisation (DPO) is widely used for safety alignment in large language models. However, prior work shows it is brittle and exhibits poor out-of-distribution (OOD) generalisation. In this paper, we investigate whether Curriculum Learning can improve the robustness of DPO-based safety alignment. We propose Staged-Competence, a curriculum-based framework that organises preference data by difficulty, employs competence-based sampling, and progressively updates the reference model during training. Averaged across three model families, Staged-Competence reduces OOD harmful response rates by 16% and jailbreak attack success rates by 20%, while preserving general capabilities with near-zero over-refusal. We further show that Staged-Competence (1) matches baseline safety with only 75% of the training data and (2) yields better separation between safe and unsafe responses. Staged-Competence is agnostic to the policy optimisation loss and can extend to other DPO variants and alignment domains. Our code and data are available at https://github.com/Sandeep5500/curriculum-learning-for-safety.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。