用多视角+分步奖励,让机器人学会精准抓取。
Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulation
- 基于多视角输入和分步离散化建模,理解操作过程
- 单次示范后仅需150次交互(约1小时)达95%成功率
- 避免语义陷阱,适合需要高精度控制的机器人任务
强化学习在真实机器人应用中的主要障碍在于有效奖励函数的设计。尽管基于学习的流程奖励模型(PRMs)前景广阔,但常受两大限制:奖励模型缺乏步骤感知能力且依赖单视角感知,导致对精细操作进展评估不可靠;奖励构造方法理论不严谨,易引发语义陷阱,误导策略优化。为此,我们提出Dopamine-Reward,一种从多视角输入中学习通用、步骤感知流程奖励模型的新方法。核心是基于3400+小时数据集训练的通用奖励模型(GRM),采用分步奖励离散化实现结构理解,通过多视角奖励融合克服感知局限。在此基础上,我们提出Dopamine-RL,一种理论稳健的策略学习框架,使用策略不变奖励重塑方法,使智能体可利用密集奖励高效自我提升,同时不改变最优策略,从根本上规避语义陷阱。大量模拟与真实世界任务实验验证了该方法的有效性。GRM在奖励评估上达到当前最佳准确率,基于GRM构建的Dopamine-RL显著提升策略学习效率。例如,将GRM一次性适配新任务(仅需单条专家轨迹),生成的奖励模型使Dopamine-RL可在150次在线滚动(约1小时真实机器人交互)内将策略成功率从接近零提升至95%,并保持良好跨任务泛化能力。
原文摘要 · Abstract (English)
The primary obstacle for applying reinforcement learning (RL) to real-world robotics is the design of effective reward functions. While recently learning-based Process Reward Models (PRMs) are a promising direction, they are often hindered by two fundamental limitations: their reward models lack step-aware understanding and rely on single-view perception, leading to unreliable assessments of fine-grained manipulation progress; and their reward shaping procedures are theoretically unsound, often inducing a semantic trap that misguides policy optimization. To address these, we introduce Dopamine-Reward, a novel reward modeling method for learning a general-purpose, step-aware process reward model from multi-view inputs. At its core is our General Reward Model (GRM), trained on a vast 3,400+ hour dataset, which leverages Step-wise Reward Discretization for structural understanding and Multi-Perspective Reward Fusion to overcome perceptual limitations. Building upon Dopamine-Reward, we propose Dopamine-RL, a robust policy learning framework that employs a theoretically-sound Policy-Invariant Reward Shaping method, which enables the agent to leverage dense rewards for efficient self-improvement without altering the optimal policy, thereby fundamentally avoiding the semantic trap. Extensive experiments across diverse simulated and real-world tasks validate our approach. GRM achieves state-of-the-art accuracy in reward assessment, and Dopamine-RL built on GRM significantly improves policy learning efficiency. For instance, after GRM is adapted to a new task in a one-shot manner from a single expert trajectory, the resulting reward model enables Dopamine-RL to improve the policy from near-zero to 95% success with only 150 online rollouts (approximately 1 hour of real robot interaction), while retaining strong generalization across tasks. Project website: https://robo-dopamine.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。