分离奖励分配与参数更新，提升强化学习稳定性
Target Policy Optimization
- 先构造目标分布，再用交叉熵优化策略
- 在稀疏奖励下性能显著优于PG/PPO等方法
- 适合需要稳定训练的大型语言模型任务
在强化学习中，给定提示后，从模型采样一组完成内容并进行评分。随之产生两个问题：哪些完成应增加概率质量，以及如何调整参数实现这一变化？标准策略梯度方法同时回答这两个问题，导致更新可能因学习率、裁剪等优化器选择而过冲或欠冲。我们提出目标策略优化（Target Policy Optimization, TPO），将这两个问题解耦。给定评分完成内容后，TPO 构造目标分布 $q_i \propto p_i^{\,\mathrm{old}} \exp(u_i)$，并通过交叉熵拟合该分布。策略在采样完成内容上的损失梯度为 $p^θ - q$，一旦策略匹配目标分布即趋于零。在表格博弈、Transformer 序列任务及十亿参数大模型的 RLVR 任务中，TPO 在简单任务上表现与 PG、PPO、GRPO、DG 相当，但在稀疏奖励下显著超越它们。代码已开源：https://github.com/JeanKaddour/tpo。
原文摘要 · Abstract (English)
In RL, given a prompt, we sample a group of completions from a model and score them. Two questions follow: which completions should gain probability mass, and how should the parameters move to realize that change? Standard policy-gradient methods answer both at once, so the update can overshoot or undershoot depending on the learning rate, clipping, and other optimizer choices. We introduce \emph{Target Policy Optimization} (TPO), which separates the two questions. Given scored completions, TPO constructs a target distribution $q_i \propto p_i^{\,\mathrm{old}} \exp(u_i)$ and fits the policy to it by cross-entropy. The loss gradient on sampled-completion logits is $p^θ- q$, which vanishes once the policy matches the target. On tabular bandits, transformer sequence tasks, and billion-parameter LLM RLVR, TPO matches PG, PPO, GRPO, and DG on easy tasks and substantially outperforms them under sparse reward. Code is available at https://github.com/JeanKaddour/tpo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。