arXiv:2603.16978cs.ROcs.LG2026-03被引 2

用视觉模型预测机器人任务的密集奖励,无需仿真专用状态信息。

Rewarding DINO: Predicting Dense Rewards with Vision Foundation Models

  • 基于语言条件与视觉基础模型学习真实奖励函数,不依赖特定轨迹。
  • 在24个Meta-World+任务上训练,模拟与真实场景均表现良好。
  • 模型小巧可直接替代传统奖励函数,适合部署到实际机器人系统。

在机器人操作中,设计良好的密集奖励不仅指示任务是否完成,还编码过程中的进展。然而,设计密集奖励通常需要仿真中才有的特权状态信息,这在真实世界实验中无法获取。因此,从摄像头图像推断任务状态信息的奖励预测模型颇具吸引力。常见方法基于专家示范,通过视觉相似性或帧序预测奖励,但会偏向特定解法,且对未覆盖状态无定义。本文提出Rewarding DINO,一种语言条件的奖励建模方法,学习实际奖励函数而非特定轨迹。模型结构紧凑,可作为分析奖励函数的直接替代,计算开销低。我们在24个Meta-World+任务数据上使用基于排名的损失进行训练,并评估成对准确性、排名相关性和校准性。Rewarding DINO在训练任务中表现优异,且在模拟和真实世界新环境中具有泛化能力,表明其学习了任务语义。我们还使用现成强化学习算法测试该模型在训练集任务上的求解能力。

原文摘要 · Abstract (English)

Well-designed dense reward functions in robot manipulation not only indicate whether a task is completed but also encode progress along the way. Generally, designing dense rewards is challenging and usually requires access to privileged state information available only in simulation, not in real-world experiments. This makes reward prediction models that infer task state information from camera images attractive. A common approach is to predict rewards from expert demonstrations based on visual similarity or sequential frame ordering. However, this biases the resulting reward function towards a specific solution and leaves it undefined in states not covered by the demonstrations. In this work, we introduce Rewarding DINO, a method for language-conditioned reward modeling that learns actual reward functions rather than specific trajectories. The model's compact size allows it to serve as a direct replacement for analytical reward functions with comparatively low computational overhead. We train our model on data sampled from 24 Meta-World+ tasks using a rank-based loss and evaluate pairwise accuracy, rank correlation, and calibration. Rewarding DINO achieves competitive performance in tasks from the training set and generalizes to new settings in simulation and the real world, indicating that it learns task semantics. We also test the model with off-the-shelf reinforcement learning algorithms to solve tasks from our Meta-World+ training set.

机器人奖励学习视觉模型强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。