用视频语言模型做唯一奖励，让机器人零样本学会新任务。
SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning
- 用视频+语言目标进行时空链式思考，生成每帧的任务进度奖励。
- 在4个仿真环境和真实机器人上实现零样本在线强化学习，24个未见任务全成功。
- 比多个强基线模型更抗奖励欺骗，适合无标注、无演示的机器人学习场景。
视觉语言模型在多种任务中表现出色，促使人们尝试将其用于机器人学习的监督。然而，当前最强模型在部分可观测性和分布偏移下表现不佳，导致策略利用感知错误而非真正完成任务。我们提出 SOLE-R1（自观察学习者），一个专为在线强化学习设计的视频-语言推理模型，作为唯一的奖励信号。仅需原始视频观测和自然语言目标，SOLE-R1 即可进行每帧的时空链式思考（CoT），生成密集的任务进度估计，直接用作奖励。为训练该模型，我们构建了大规模视频轨迹与推理合成管道，生成与连续进展监督对齐的时间锚定 CoT 轨迹。结合基础的空间与多帧时序推理，采用监督微调与可验证奖励强化学习相结合的混合框架进行训练。在四个不同仿真环境及真实机器人设置中，SOLE-R1 实现了从随机初始化开始的零样本在线强化学习：机器人无需真值奖励、成功指标、示范或任务特定调优，即可学习未见过的操作任务。在24个未见任务中全部成功，显著优于 Robometer、RoboReward、ReWiND、GPT-5 和 Gemini-3-Pro 等强基线模型，并展现出更强的抗奖励劫持能力。所有模型、数据、代码与演示已发布于匿名页面：https://philip-mit.github.io/sole-r1/
原文摘要 · Abstract (English)
Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today's strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. We introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including Robometer, RoboReward, ReWiND, GPT-5, and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking. We release all models, data, code, and demos at the anonymous page: https://philip-mit.github.io/sole-r1/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。