用少量示范优化视觉语言模型的奖励提示，提升机器人学习准确性。
From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models
- 基于3-10条示范轨迹，在测试时优化奖励模型的语言指令。
- 显著降低假阳性率，同时保持真阳性，提升策略学习效果。
- 无需额外训练，适合真实机器人场景的零样本奖励设计。
强化学习依赖精确的奖励函数，但现实应用如机器人中常缺乏或难以手工设计。近期研究尝试利用预训练视觉语言模型（VLM）作为零样本奖励模型，但若提示工程不当，易产生次优奖励，虚假正例会严重损害下游策略学习。在机器人任务中，通常可获取少量专家示范数据，可用于策略学习前优化奖励模型。本文提出Demo2Reward：一种测试时适应技术，基于3-10条轨迹示范，优化奖励模型的语言指令，以减少假阳性并保留真阳性。该方法无需额外模型训练或计算资源即可应用于策略学习。实验表明，Demo2Reward在多种模拟机器人任务和策略骨干上均优于现有零样本与少样本VLM奖励模型。最后，我们验证其在真实机器人场景的有效性，实现无需人工设计奖励函数的策略学习。
原文摘要 · Abstract (English)
Reinforcement learning relies on accurate reward functions, which are often hand-crafted or even unavailable in real-world applications, such as robotics. Recent work has explored the zero-shot reasoning capabilities of pre-trained Vision-Language Models (VLMs) as reward models. However, without careful prompt engineering, these approaches tend to produce suboptimal rewards, where false positive predictions can severely degrade downstream policy learning. In robotics, limited datasets comprising expert demonstrations are often collected to bootstrap policy learning. This scenario provides an opportunity to optimize a reward model prior policy training. We propose Demo2Reward a test-time adaptation technique to optimize the language instruction of a reward model based on a few demonstrations (3-10 trajectories) to reduce false positives while preserving true positives. Crucially, this requires no additional model training or computation resources during policy learning. We show that Demo2Reward consistently outperforms existing zero- and few-shot VLM reward models across a range of simulated robotic tasks and policy backbones. Finally, we demonstrate that Demo2Reward effectively transfers to a real-world robotic learning scenario, enabling policy learning without manually engineering a reward function.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。