用重排序提升大模型安全对齐效率，省下300倍算力。
Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling
- 通过重排偏好数据替代在线采样，降低计算开销。
- 在多个数据集上显著提升安全性，避免分布偏移问题。
- 适合追求高效安全对齐的AI研发人员使用。
大型语言模型（LLM）的安全对齐常采用强化学习方法（如直接偏好优化，DPO），但面临分布偏移挑战。现有方法通常依赖从目标策略中在线采样，消耗大量计算资源。本文提出假设：在离线策略训练中，尽管策略生成输出的排名顺序可能变化，其整体分布保持相对稳定。基于此，将目标策略采样转化为高效重排序偏好数据。利用模型自身安全判断能力提取奖励信号，并据此计算偏好标签置信度以进行重排序。大量实验与理论分析表明，该方法有效缓解分布偏移，显著提升安全性，同时避免约300倍的计算开销。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) algorithms for safety alignment of Large Language Models (LLMs), such as Direct Preference Optimization (DPO), encounter the challenge of distribution shift. Current approaches typically address this issue through online sampling from the target policy, which requires significant computational resources. In this paper, we hypothesize that during off-policy training, while the ranking order of output generated by policy changes, their overall distribution remains relatively stable. This stability allows the conversion of the sampling process from the target policy into a computationally efficient re-ranking of preference data. Building on this hypothesis, we propose a new framework that leverages the model's intrinsic safety judgment capability to extract reward signals, which are then used to calculate label confidence for preference reordering. Extensive experiments and theoretical analysis demonstrate that the proposed method effectively addresses the distribution shift issue, remarkably enhancing the safety performance while avoiding about 300x computational overheads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。