QRPO让大模型直接学绝对奖励，还能离线训练，效果更好。
Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions
- 用分位数奖励实现闭式解的强化学习，无需相对信号
- 在多个评测中超越DPO、REBEL等方法，8B模型表现最优
- 减少长度偏差，适合追求高效训练和高质量输出的研究者
将大语言模型与点对点绝对奖励对齐，以往依赖在线、同策略算法如PPO和GRPO。而更简单的离线或异策略方法（如DPO、REBEL）仅能处理偏好对或相对信号。为此，我们提出分位数奖励策略优化（QRPO），可在保留DPO类方法简单性与离线适用性的前提下，从点对点绝对奖励中学习。QRPO利用分位数奖励，使回归目标逼近带KL正则化强化学习目标的闭式解，获得可解析计算的分区函数，无需依赖相对信号抵消该项。此外，QRPO可通过增加计算量来更精确估计分位数奖励，开辟了预计算扩展的新维度。实验表明，QRPO在聊天与编程任务上持续领先：相比DPO、REBEL、SimPO，在多种数据集与8B规模模型下，均取得最优的奖励模型得分、AlpacaEval 2与LeetCode成绩。最后，我们发现使用稳健奖励而非转换为偏好进行训练，可有效降低长度偏差。
原文摘要 · Abstract (English)
Aligning large language models with pointwise absolute rewards has so far required online, on-policy algorithms such as PPO and GRPO. In contrast, simpler methods that can leverage offline or off-policy data, such as DPO and REBEL, are limited to learning from preference pairs or relative signals. To bridge this gap, we introduce Quantile Reward Policy Optimization (QRPO), which learns from pointwise absolute rewards while preserving the simplicity and offline applicability of DPO-like methods. QRPO uses quantile rewards to enable regression to the closed-form solution of the KL-regularized RL objective. This reward yields an analytically tractable partition function, removing the need for relative signals to cancel this term. Moreover, QRPO scales with increased compute to estimate quantile rewards, opening a new dimension for pre-computation scaling. Empirically, QRPO consistently achieves top performance on chat and coding evaluations--reward model scores, AlpacaEval 2, and LeetCode--compared to DPO, REBEL, and SimPO across diverse datasets and 8B-scale models. Finally, we find that training with robust rewards instead of converting them to preferences induces less length bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。