不依赖价值网络，用回溯树实现细粒度动作奖励分配。
RTMC: Step-Level Credit Assignment via Rollout Trees
- 通过回溯树聚合共享状态的多步轨迹回报，生成每步动作价值。
- 在SWE-bench Verified上比GRPO提升3.2个百分点的通过率。
- 无需训练网络，适合稀疏奖励场景下的智能体优化。
多步智能体强化学习依赖细粒度的信用分配，但现有方法选择有限：无价值网络的方法如GRPO对轨迹中每个动作赋予相同优势，而学习型价值网络则带来显著开销且在稀疏奖励下易失效。我们观察到，针对同一问题的多组回溯轨迹常经过重叠的中间状态，隐含形成一棵分支在决策点逐步分离的树。基于此，我们提出回溯树蒙特卡洛（RTMC）优势估计方法，通过聚合共享同一状态的回溯轨迹的回报统计，生成每步的Q值和优势——无需任何学习型批评者。状态-动作签名系统将原始交互历史压缩为紧凑可比的表示，使跨回溯状态匹配成为可能。在SWE-bench Verified上，RTMC相比GRPO将pass@1提升3.2个百分点。
原文摘要 · Abstract (English)
Multi-step agentic reinforcement learning benefits from fine-grained credit assignment, yet existing approaches offer limited options: critic-free methods like GRPO assign a uniform advantage to every action in a trajectory, while learned value networks introduce notable overhead and can be fragile under sparse rewards. We observe that group rollouts targeting the same problem often traverse overlapping intermediate states, implicitly forming a tree whose branches diverge at successive decision points. Building on this insight, we introduce Rollout-Tree Monte Carlo (RTMC) advantage estimation, which aggregates return statistics across rollouts sharing a common state to produce per-step Q-values and advantages--without any learned critic. A state-action signature system compresses raw interaction histories into compact, comparable representations, making cross-rollout state matching tractable. On SWE-bench Verified, RTMC improves pass@1 by 3.2 percentage points over GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。