不依赖价值函数的奖励优化方法，提升生成质量与稳定性
Value-Free Policy Optimization via Reward Partitioning
- 通过提示级奖励分布划分实现奖励归一化，避免价值函数估计
- 在多个模型上优于SFT、KTO和DRO，生成更对齐、多样且低毒性
- 适合追求高效、稳定且无需额外训练模块的RLHF应用
单轨迹偏好优化方法从((提示, 回应, 奖励))元组数据集中学习，通过直接利用标量反馈提供了一种实用的配对偏好学习替代方案。现有方法如直接奖励优化（DRO）虽表现良好，但依赖价值函数估计，引入额外方差、优化复杂性和对离策略数据的敏感性。本文提出奖励划分优化（RPO），一种简单且可扩展的奖励驱动目标，完全消除对价值函数学习的需求。RPO通过基于划分的公式直接从提示级奖励分布估计归一化奖励，生成无需辅助模型或强化学习循环的稳定监督优化目标。我们在多个编码器-解码器和仅解码器语言模型上使用自动指标、大模型作为评判者评估及优化稳定性分析进行了评估。实验结果表明，RPO在多个方面持续优于强基线方法（包括SFT、KTO和DRO），生成更对齐、更多样且毒性更低的内容。
原文摘要 · Abstract (English)
Single-trajectory preference optimization methods learn from datasets of ((prompt, response, reward)) tuples, offering a practical alternative to pairwise preference learning by directly leveraging scalar feedback. Existing approaches such as Direct Reward Optimization (DRO) have demonstrated promising results but rely on value function estimation, introducing additional variance, optimization complexity, and sensitivity to off-policy data. We introduce Reward Partition Optimization (RPO), a simple and scalable reward-driven objective that eliminates the need for value function learning. RPO normalizes rewards through a partition-based formulation estimated directly from prompt-level reward distributions, yielding a stable supervised optimization objective without auxiliary models or reinforcement learning loops. We evaluate RPO across multiple encoder-decoder and decoder-only language models using automatic metrics, LLM-as-a-judge evaluations, and optimization stability analyses. Experimental results show that RPO consistently outperforms strong baselines, including SFT, KTO, and DRO, while producing more aligned, diverse, and less toxic generations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。