用分布优化提升奖励学习后的策略性能，更高效且更稳定。
DDO-RM: Distribution-Level Policy Improvement after Reward Learning
- 将奖励得分转化为目标分布，通过镜面下降优化策略
- 在Pythia-410M上对偶准确率提升至0.56，均值差增至0.53
- 适合追求高精度和样本效率的强化学习研究者
近期理论表明，当奖励函数比生成策略更简单时，先训练奖励模型的方法比直接拟合策略更节省样本。我们提出DDO-RM，一种基于有限候选集的分布级决策优化方法，将奖励分数转换为显式目标分布。与PPO-based RLHF或DPO不同，DDO-RM采用KL正则化的镜面下降更新，将策略投影到奖励优化后的分布上。在Pythia-410M上的初步实验显示，DDO-RM在对偶准确率（0.52→0.56）和平均边际（0.13→0.53）上优于DPO。该框架为奖励学习与镜面下降策略改进提供了严谨的理论联系。
原文摘要 · Abstract (English)
Recent theory suggests that reward-model-first methods can be more sample-efficient than direct policy fitting when the reward function is statistically simpler than the induced policy. We propose DDO-RM, a finite-candidate decision-optimization method that converts reward scores into an explicit target distribution. Unlike PPO-based RLHF or DPO, DDO-RM performs a KL-regularized mirror-descent update to project the policy toward a reward-improved distribution over a candidate set. Preliminary experiments on Pythia-410M show that DDO-RM outperforms DPO in pair accuracy (0.52 to 0.56) and mean margin (0.13 to 0.53). Our framework provides a principled connection between reward learning and mirror-descent policy improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。