用树结构精准分配强化学习中长序列的奖励，提升大模型推理能力。
Exploiting Tree Structure for Credit Assignment in RL Training of LLMs
- 将多个回复转为前缀树，非参数化计算每步前缀的价值。
- 在分支点引入时序差分修正，实现无需价值网络的精准奖励分配。
- 相比现有方法更快更准，适合数学与医疗问答等可验证任务。
强化学习能提升大模型的推理能力，但长序列中稀疏且延迟的奖励使得逐标记的信用分配成为关键瓶颈。本文研究可验证奖励场景，即每个提示可生成多个回复，最终答案可被验证。数学与医疗问答任务符合此设定，仅少数决策标记影响结果。PPO虽支持逐标记优势，但需同时训练策略与价值网络，复杂且易过拟合；GRPO无价值网络，支持可验证奖励，但将整体回报平均分配给所有标记并忽略分支结构。为此，本文提出前缀树（Prefix-to-Tree, P2T）方法，将一组回复转化为前缀树，并通过聚合后代结果非参数化计算前缀值𝑉(𝑠)。基于P2T，提出TEMPO（Tree-Estimated Mean Prefix Value for Policy Optimization），一种无价值网络算法，在GRPO的组间相对信号基础上，引入由树结构导出的分支门控时序差分修正项。在非分支点，时序差分项为零，TEMPO退化为GRPO；在分支点，提供精确的逐标记信用分配，无需额外教师或训练价值网络。在Qwen3-1.7B/4B上，TEMPO在MATH、MedQA等分布内及GSM-HARD、AMC23、MedMCQA、MMLU-Medical等分布外基准上均优于PPO与GRPO，达到更高验证准确率，且耗时相近。
原文摘要 · Abstract (English)
Reinforcement learning improves LLM reasoning, yet sparse delayed reward over long sequences makes token-level credit assignment the key bottleneck. We study the verifiable-reward setting, where the final answer is checkable and multiple responses can be drawn per prompt. Reasoning tasks in math and medical QA align with this setup, where only a few decision tokens significantly impact the outcome. PPO offers token-level advantages with a learned value model, but it is complex to train both the actor and critic models simultaneously, and it is not easily generalizable, as the token-level values from the critic model can make training prone to overfitting. GRPO is critic-free and supports verifiable rewards, but spreads a single sequence-level return across tokens and ignores branching. We introduce \textbf{Prefix-to-Tree (P2T)}, a simple procedure that converts a group of responses into a prefix tree and computes \emph{nonparametric} prefix values \(V(s)\) by aggregating descendant outcomes. Built on P2T, we propose \textbf{TEMPO} (\emph{\textbf{T}ree-\textbf{E}stimated \textbf{M}ean Prefix Value for \textbf{P}olicy \textbf{O}ptimization}), a critic-free algorithm that augments the group-relative outcome signal of GRPO with \emph{branch-gated} temporal-difference corrections derived from the tree. At non-branch tokens, the temporal-difference (TD) term is zero, so TEMPO reduces to GRPO; at branching tokens, it supplies precise token-level credit without a learned value network or extra judges/teachers. On Qwen3-1.7B/4B, TEMPO outperforms PPO and GRPO on in-distribution (MATH, MedQA) and out-of-distribution (GSM-HARD, AMC23, MedMCQA, MMLU-Medical) benchmarks, and reaches higher validation accuracy with roughly the same wall-clock time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。