用树结构优化群体强化学习，让模型更高效地思考。
TreeAdv: Tree-Structured Advantage Redistribution for Group-Based RL
- 构建树形轨迹森林，高不确定处分支，低不确定处共享
- 重新分配完整轨迹优势，提升内部节点的奖励精度
- 在10个数学推理任务中减少生成token数，效果更优
基于群体目标的强化学习（如GRPO）常用于对齐大语言模型在复杂推理任务上的表现。但传统GRPO将每条轨迹视为独立扁平序列，对所有标记分配单一序列级优势，导致样本效率低，并偏向冗长、重复的思维链，无法提升逻辑深度。本文提出TreeAdv（树结构优势重分配），显式建模群体轨迹的树状结构，用于探索与优势分配。具体而言，TreeAdv通过熵驱动采样构建一组树（森林），在高不确定性决策点分支，而低不确定性标记在不同轨迹间共享。随后，通过重新分配完整轨迹（所有叶节点）的优势，聚合内部树段的标记级优势，并可轻松适配GRPO或GSPO等群体目标。在10个数学推理基准上，TreeAdv持续优于GRPO和GSPO，且在相同监督、数据和解码预算下，生成的标记数量显著减少。
原文摘要 · Abstract (English)
Reinforcement learning with group-based objectives, such as Group Relative Policy Optimization (GRPO), is a common framework for aligning large language models on complex reasoning tasks. However, standard GRPO treats each rollout trajectory as an independent flat sequence and assigns a single sequence-level advantage to all tokens, which leads to sample inefficiency and a length bias toward verbose, redundant chains of thought without improving logical depth. We introduce TreeAdv (Tree-Structured Advantage Redistribution for Group-Based RL), which makes the tree structure of group rollouts explicit for both exploration and advantage assignment. Specifically, TreeAdv builds a group of trees (a forest) based on an entropy-driven sampling method where each tree branches at high-uncertainty decisions while sharing low-uncertainty tokens across rollouts. Then, TreeAdv aggregates token-level advantages for internal tree segments by redistributing the advantages of complete rollouts (all leaf nodes), and TreeAdv can easily apply to group-based objectives such as GRPO or GSPO. Across 10 math reasoning benchmarks, TreeAdv consistently outperforms GRPO and GSPO, while using substantially fewer generated tokens under identical supervision, data, and decoding budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。