通过动态调控提升安全对齐的鲁棒性，解决偏好训练中的噪声敏感问题。
CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment
- 利用隐式奖励边界作为训练时导航，控制更新方向与幅度。
- 在四个模型上显著提升对抗标签翻转噪声的稳定性，优于主流DPO方法。
- 无需额外奖励模型或监督，适合高风险场景下的安全对齐应用。
直接偏好优化（DPO）已成为安全对齐的标准框架，但其依赖成对偏好更新的特点使其对不完美监督敏感。现有鲁棒DPO方法多通过全局损失修正或外部数据干预来缓解，却忽视了不可靠比较对批次优化动态的破坏。本文提出CompassDPO，一种无奖励DPO框架，通过动态控制稳定偏好优化。利用隐式DPO奖励边际作为训练时的导航信号，从更新方向和幅度两方面调节样本影响：方向控制采用稀疏、预算限制且延迟启动的损失混合，抑制与偏好趋势冲突的更新分量；幅度控制则自适应地软截尾高损失尾部贡献，降低极端样本主导性的同时保留硬样本有用梯度。两项机制均仅使用标准DPO训练中已有信号，无需外部奖励模型或额外监督。在PKU-SafeRLHF数据集上，针对四种骨干模型及多个分布外安全基准的实验表明，CompassDPO在标签翻转噪声下仍持续优于原始DPO和强基线方法，展现更强鲁棒性。代码已公开于https://anonymous.4open.science/r/CompassDPO-4D00。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) has become a standard framework for safety alignment, but its reliance on pairwise preference updates makes training sensitive to imperfect supervision. Existing robust DPO methods often address this sensitivity through global loss corrections or external data-level interventions, while largely overlooking how unreliable comparisons distort batch-level optimization dynamics. We propose CompassDPO, a reward-free DPO framework that stabilizes preference optimization through dynamics control. Using the implicit DPO reward margin as a training-time compass, CompassDPO regulates sample influence along two complementary axes: update direction and update magnitude. For directional control, it applies sparse, budgeted, and warm-up delayed loss mixing to attenuate update components that conflict with the emerging preference direction. For magnitude control, it adaptively soft-winsorizes high-loss tail contributions, reducing tail dominance while preserving useful gradients from hard examples. Both mechanisms use only signals available during standard DPO training and require no external reward model or additional supervision. Experiments on PKU-SafeRLHF across four backbones and multiple out-of-distribution safety benchmarks show that CompassDPO consistently improves robustness over vanilla DPO and strong DPO-family baselines, especially under controlled label-flip noise. Code is available at https://anonymous.4open.science/r/CompassDPO-4D00
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。