用视频语言模型的词概率做机器人任务进度奖励,无需训练
TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics
- 通过预训练视觉语言模型的词概率探测任务完成度
- 在130个真实任务上比现有无训练方法提升显著
- 适合需要密集反馈的机器人学习场景
通用机器人学习需要稠密、指令相关的反馈来区分有意义的任务进展与停滞或失败行为。然而,大规模获取此类反馈仍具挑战,因现有方法常依赖人工进度标注、特定任务示范或在精选机器人数据集上训练的奖励模型。本文提出TOPReward,一种无需训练的进度奖励方法,通过探测预训练视频-语言模型(VLM)内部的词概率,而非要求其生成数值进度值。给定视频前缀和语言指令,TOPReward衡量模型认为任务已完成的似然性,将隐含的视频-语言理解转化为稠密奖励信号,无需任务特定奖励模型训练或人工标注的进度标签。我们在自建的真实世界操作基准ManiRewardBench(涵盖130个独特任务及4种机器人平台)和Open X-Embodiment数据集上评估该方法。在这些设置下,TOPReward在开源模型上显著优于先前的无训练VLM奖励方法,并在进度估计指标上与训练过的奖励模型基线相当,且无需奖励模型训练。额外分析表明,该奖励对指定指令敏感,非仅由时间索引解释。此外,TOPReward支持成功检测和离线奖励加权行为克隆等下游应用。
原文摘要 · Abstract (English)
General-purpose robot learning requires dense, instruction-conditioned feedback that can distinguish meaningful task progress from stalled, failed, or partially completed behavior. Yet obtaining such feedback at scale remains difficult, since existing approaches often rely on manual progress annotations, task-specific demonstrations, or reward models trained on curated robot datasets. We introduce TOPReward, a training-free progress reward method that probes pretrained Video-Language Models (VLMs) through their internal token probabilities rather than asking them to generate numerical progress values. Given a video prefix and a language instruction, TOPReward measures the model's likelihood that the instructed task has been completed, converting latent video-language understanding into a dense reward signal without task-specific reward-model training or manually annotated progress labels. We evaluate TOPReward on ManiRewardBench, our real-world manipulation benchmark spanning 130 unique tasks and four robot platforms, as well as on Open X-Embodiment datasets. Across these settings, TOPReward substantially outperforms prior training-free VLM reward methods on open-source models and is competitive with a trained reward-model baseline on progress-estimation metrics, while requiring no reward-model training. Additional analyses show that the reward is sensitive to the specified instruction and is not explained by time index alone. Finally, TOPReward supports downstream applications including success detection and offline reward-weighted behavior cloning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。