用能量模型提升语言模型对齐的鲁棒性与泛化能力
Energy-Based Reward Models for Robust Language Model Alignment
- 通过显式建模奖励分布，捕捉人类偏好不确定性
- 在安全关键任务中比标准奖励模型提升5.97%对齐效果
- 无需重训练即可优化现有模型，适合快速部署
奖励模型（RMs）是将大语言模型（LLMs）与人类偏好对齐的关键。然而，它们常难以捕捉复杂的人类偏好，且泛化能力弱。为此，我们提出能量基奖励模型（EBRM），一种轻量级后处理优化框架，可增强奖励模型的鲁棒性与泛化性。EBRM 显式建模奖励分布，捕捉人类偏好的不确定性，并减轻噪声或错标标注的影响。其通过冲突感知数据过滤、标签噪声感知对比学习和混合初始化实现优化。值得注意的是，EBRM 不需重新训练即可提升现有模型性能，计算高效且适用于多种模型与任务。在多个奖励模型基准上的实证评估显示，其显著提升了鲁棒性与泛化性，在安全关键对齐任务中最高提升5.97%。强化学习实验进一步证实，优化后的奖励能有效延迟奖励黑客行为，提升对齐质量。该方法为现有对齐流程提供了一种可扩展、高效的增强方案。代码已开源。
原文摘要 · Abstract (English)
Reward models (RMs) are essential for aligning Large Language Models (LLMs) with human preferences. However, they often struggle with capturing complex human preferences and generalizing to unseen data. To address these challenges, we introduce Energy-Based Reward Model (EBRM), a lightweight post-hoc refinement framework that enhances RM robustness and generalization. EBRM models the reward distribution explicitly, capturing uncertainty in human preferences and mitigating the impact of noisy or misaligned annotations. It achieves this through conflict-aware data filtering, label-noise-aware contrastive training, and hybrid initialization. Notably, EBRM enhances RMs without retraining, making it computationally efficient and adaptable across different models and tasks. Empirical evaluations on RM benchmarks demonstrate significant improvements in both robustness and generalization, achieving up to a 5.97% improvement in safety-critical alignment tasks compared to standard RMs. Furthermore, reinforcement learning experiments confirm that our refined rewards enhance alignment quality, effectively delaying reward hacking. These results demonstrate our approach as a scalable and effective enhancement for existing RMs and alignment pipelines. The code is available at EBRM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。