arXiv:2608.15680cs.ROcs.AI2026-08

让机器人更懂何时该继续、何时该重来,提升复杂操作的稳定性。

Robo-Dopamine 2.0: History-Conditioned and OOD-Aware Process Reward Modeling for Robotic Manipulation

论文配图:Robo-Dopamine 2.0: History-Conditioned and OOD-Aware Process Reward Modeling for Robotic Manipulation
图 1 · 摘自论文原文
  • 用历史轨迹和合成异常数据判断动作是否合理,避免误判失败。
  • 在40万次奖励评估下,任务成功率提升至86.8%,真实世界插入成功71/80次。
  • 适合需要长期稳定执行的机器人操控场景,尤其应对意外干扰。

视觉-语言-动作(VLA)模型虽提升机器人操作能力,但仍易受误差累积、环境变化和偏离轨迹状态影响。强化学习可优化预训练策略,但稀疏的成功信号限制探索,而人工设计的密集奖励成本高且任务特定。现有视觉奖励模型多依赖静态前后观察,导致时间歧义,并难以区分鲁棒性变化与任务失效。本文提出Robo-Dopamine 2.0,一种历史感知且对分布外(OOD)敏感的过程奖励模型,采用成对预测接口。其结合:(1) 历史条件化的成对奖励,利用源对齐参考图生成合成OOD查询,同时使用实际执行轨迹进行在线查询,保持目标终点一致;(2) 带符号的进度空间,明确表示有效进展、鲁棒性、失败与恢复状态。引入带转移感知回放的符号跳跃课程,先学习粗粒度执行顺序,再精细校准进展。构建了首个OOD轨迹数据集及五家族基准。参考图使平均视觉顺序一致性(VOC)从0.967提升至0.986,对OOD鲁棒性VOC从0.906提升至0.958。相同40万次奖励预算下,符号跳跃训练(25%回放)达到0.9872均值VOC,优于匹配池随机控制的0.9858。下游强化学习中,完整模型实现86.8%平均RoboTwin成功率,真实世界插入成功71/80次。

原文摘要 · Abstract (English)

Vision-language-action (VLA) models improve robotic manipulation but remain vulnerable to compounding errors, scene changes, and off-trajectory states. Reinforcement learning can refine pretrained VLA policies, yet sparse success signals hinder exploration, while engineered dense rewards are costly and task-specific. Existing learned visual reward models often rely on static before-after observations, causing temporal ambiguity and weak discrimination between robustness-preserving variations and task-invalid failures under out-of-distribution (OOD) execution. We introduce Robo-Dopamine 2.0, a history- and OOD-aware process reward model with a pairwise prediction interface. It combines (1) history-conditioned pairwise rewards that use source-aligned reference panels for synthetic OOD queries and observed rollout history for online queries, while preserving the queried endpoints, and (2) an OOD-aware signed progress space that represents valid progress, robustness, failure, and recovery. A Signed-Hop Curriculum with transition-aware replay learns coarse execution ordering before fine-grained progress calibration. We also construct an OOD trajectory dataset and a five-family benchmark. Reference panels improve mean visual order consistency (VOC) from 0.967 to 0.986 and OOD-robust VOC from 0.906 to 0.958. With the same 400K pairwise-reward budget, Signed-Hop training with 25% replay reaches 0.9872 mean VOC, compared with 0.9858 for a matched-pool shuffled control. In downstream reinforcement learning, the full model achieves 86.8% mean RoboTwin success and 71/80 successful real-world insertions.

机器人操作强化学习奖励建模分布外检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。