通过预估后续任务成功率，提升长序列机器人装配的稳定性。
Foresight Residual RL for Long-Horizon Robot Manipulation with Vision-Language-Action Models
- 在基础视觉语言动作模型上，用离线预测未来成功概率来优化每步交接质量。
- 在三个阶段的拧螺母任务中，全任务成功率提升至85.6%，远超基线的54.5%。
- 适合需要高精度、多步骤操作的机器人装配场景，尤其关注状态衔接可靠性。
视觉-语言-动作（VLA）策略具备强泛化操作先验，但在高精度、接触密集型装配任务中常因长时序信用分配和子任务耦合而失败：当前技能的几何成功状态可能对后续技能极为脆弱。我们发现在冻结的VLA基底策略上进行残差强化学习（RL）时，仅靠稀疏的成功奖励虽能独立提升各子任务表现，但串联后几乎无增益，因终态质量未受控制。为此提出前瞻残差强化学习（Foresight Residual RL），通过为每个子任务的稀疏成功奖励叠加离线估计的前瞻值（即当前子任务终态下未来子任务成功的概率）来优化交接质量。具体而言，(i) 利用基底策略终态图像与下游回放统计标签训练视觉前瞻预测器；(ii) 通过反向前瞻诱导训练残差策略，以预测输出作为奖励乘数。在Isaac Gym中的三阶段扳手拧螺母任务（抓取、移动插入、旋转）上，本方法实现85.6%的全任务成功率，显著优于标准残差RL（54.5%）和VLA基线，同时保持各子任务成功率不变。结果表明，提升长时序性能的关键在于塑造每一步生成的成功状态，而不仅是是否达成成功。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) policies offer strong general-purpose manipulation priors, but often fail on tight-tolerance, contact-rich assembly due to long-horizon credit assignment and subtask coupling: a state that is geometrically successful for the current skill can be brittle for downstream skills. We show this failure mode in residual reinforcement learning (RL) over a frozen VLA base policy: constant sparse success rewards improve each subtask in isolation yet yield little or no gain when skills are chained, because terminal state quality is uncontrolled. We propose Foresight Residual RL, which optimizes handoff quality by augmenting each subtask's sparse success reward with an offline-estimated foresight value -- the probability of future subtask success conditioned on the terminal state of the current subtask. Concretely, we (i) train a visual foresight predictor from images of terminal states of the base policy, labeled using downstream rollout statistics, and (ii) train residual policies via backward foresight induction, using the predictor output as a reward multiplier. On a three-phase wrench-based nut-tightening assembly task in Isaac Gym (grasp, move-insert, rotate), our method achieves 85.6% full-task success, outperforming standard subtask residual RL (54.5%) and VLA baselines, while leaving per-subtask success unchanged. These results highlight that improving long-horizon performance requires shaping which successful states are produced at each sub-task, not only whether success occurs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。