让机器人从失败中学习，用语言重定义任务提升训练效率
Learning More from Less: Reinforcement Learning from Hindsight
- 用语言重标注失败轨迹，将失败转化为新任务进行训练
- 在未见任务上实现5倍样本效率提升，超越密集奖励基线
- 适用于多种视觉语言动作模型，物理机器人实测有效
强化学习（RL）越来越多用于后训练视觉-语言-动作（VLA）模型，但每次更新都需消耗昂贵且耗时的机器人轨迹采集，因此样本效率至关重要。操控任务通常仅提供稀疏奖励，导致弱策略在训练初期几乎每次都过早失败，难以从中学习，即使这些失败行为本身具有连贯性。然而，一次失败实际上可能是另一项任务的成功。本文提出「从后见中学习」（LfH），将事后重标注引入VLA的强化学习后训练中，通过评估失败轨迹实际完成的任务来重标注指令和奖励。一个统一的视觉语言模型同时重标注指令与奖励，为一组失败轨迹生成事后指令，并评分每个轨迹满足该指令的程度，随后策略联合训练原始与重标注轨迹。由于VLA具备语言泛化能力，基于语言的重标注使同一轨迹能学习更多内容。在分布外的LIBERO-PRO任务上，标准RL改善缓慢，而LfH实现5倍样本效率提升，优于密集进度奖励基线，效果在多种VLA主干网络及真实Franka机器人上均成立。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, but every update consumes robot rollouts that are slow and costly to collect, making sample efficiency a central concern. Manipulation tasks typically provide only sparse rewards, so a weak policy fails almost every rollout early in training and has little to learn from, even when those failures execute coherent behavior. Such a failure, however, is a success at a different task. We present Learning from Hindsight (LfH), which brings hindsight relabeling to RL post-training of VLAs by scoring failed rollouts against the tasks they actually achieved. A single vision-language model relabels both the instruction and the reward, proposing a hindsight instruction for a group of failed rollouts and scoring how well each satisfies it, and the policy trains on the relabeled and original rollouts jointly. Because VLAs generalize across language, relabeling in language lets the policy learn more from the same trajectories. On out-of-distribution LIBERO-PRO tasks, where standard RL improves only slowly, LfH achieves $5\times$ improvement in sample efficiency, and outperforms a dense progress-reward baseline. The gains hold across VLA backbones and on a physical Franka robot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。