改进视觉语言动作强化学习中的奖励分配，避免早期正确操作被失败拖累。
Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning
- 按任务阶段划分轨迹，仅在相同阶段内比较并计算奖励
- 在RoboTwin 2.0上提升任务成功率与采样效率
- 适合长序列任务，尤其擅长保留共享前置步骤的训练
以结果为导向的强化学习为从稀疏的任务成功反馈中后训练视觉语言动作（VLA）策略提供了一种可扩展的方法。在常见的基于GRPO的VLA后训练中，一个轨迹级别的优势值被应用于该轨迹中的每一步动作。若某条轨迹完成了多个有效阶段但最终失败，则其前期正确动作会被惩罚，这种现象称为轨迹级信用混淆。Temporal GRPO通过识别可检测的任务阶段，将每条轨迹对齐到对应的阶段动作区间，并仅在相同阶段间进行比较，从而获得阶段优势。这些阶段优势在单次策略更新中被应用于对应的动作区间。在RoboTwin 2.0上，Temporal GRPO显著提升了任务成功率和样本效率，且在不同任务时长下均有稳定收益。在LIBERO-Long上的受控更新实验表明，该方法能保持共享的前置阶段，并将改进集中于首个出现分歧的阶段。
原文摘要 · Abstract (English)
Outcome-driven reinforcement learning offers a scalable way to post-train vision-language-action (VLA) policies from sparse task-success feedback. In common GRPO-based VLA post-training, one rollout-level advantage is applied to every action in the trajectory. A rollout that completes several valid stages but fails later can therefore penalize the actions that produced its earlier progress. We call this trajectory-level credit aliasing. Temporal GRPO addresses this problem by constructing detectable task stages, aligning each rollout with stage-specific action intervals, and comparing only rollouts that have entered the same stage. The resulting stage advantages are applied to their corresponding intervals in a single policy update. On RoboTwin 2.0, Temporal GRPO improves task success and sample efficiency, with consistent gains across task horizons. Controlled updates on LIBERO-Long preserve shared prerequisite stages and concentrate improvement at the first stage where rollout outcomes diverge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。