用时间距离替代传统奖励,让机器人模型在海量数据上高效学习通用价值
RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance

- 以观测到目标的时间距离作为监督信号,无需偏好或进度标注
- 在7000小时、300万段视频上训练,零样本泛化至新任务与视角,性能超越现有方法
- 可直接转化为密集奖励,显著提升真实场景中机械臂操作成功率
通用奖励模型正成为机器人规模化学习的瓶颈,而从大规模异构数据中学习价值相关能力的方法仍不明确。现有方法依赖任务内锚点(如偏好或归一化进展),难以跨实体和数据源迁移。我们提出RynnValue——一个开源的机器人操作价值基础模型,将这些锚点替换为时间距离,即从观测到语言指定目标的有向代价。由于时间距离可直接从时间戳推导,RynnValue得以在超过7,000小时、约300万条指令条件片段上训练,无需偏好或进展标注。为确保大规模下时间价值学习的可靠性,我们结合随机时间采样、时间顺序打乱和价值隔离注意力,抑制对失败和退化的预测不敏感的捷径。该模型未使用偏好标签,在RBM-EVAL-OOD上达到平均Kendall's tau_a为0.675,优于完全依赖偏好的最先进水平(0.655),且远超仅依赖进展的基线(0.292);同时实现零样本跨任务、跨实体、跨视角泛化。通过基于势能的重塑转换为密集奖励后,线上策略成功率从52.5%提升至72.5%,线下从63.8%提升至82.5%。这些结果确立了时间距离作为可扩展的监督目标和通用机器人策略的实际奖励接口。
原文摘要 · Abstract (English)
General-purpose reward models are increasingly the bottleneck for scaling robot learning, yet the recipe for learning value-related capabilities from large-scale heterogeneous corpora remains underexplored. Existing approaches tie supervision to task-internal anchors such as preferences or normalized progress, none of which transfer cleanly across embodiments and data sources. We introduce RynnValue, an open-source value foundation model for robotic manipulation that replaces these anchors with temporal distance, the directed cost-to-go from an observation to the language-specified goal. Because temporal-distance labels can be derived directly from timestamps, RynnValue scales to over 7,000 hours and roughly 3M instruction-conditioned clips without preference or progress annotations. To make temporal-value learning reliable at scale, we combine random temporal sampling, temporal-order shuffling, and value-isolation attention, suppressing shortcuts that would leave predictions insensitive to failures and regressions. Trained without preference labels, RynnValue attains an average Kendall's tau_a of 0.675 on RBM-EVAL-OOD, surpassing the fully preference-supervised state of the art (0.655) and more than doubling a progress-only counterpart (0.292), while generalizing zero-shot to unseen tasks, embodiments, and viewpoints. Converted into dense rewards via potential-based shaping, it raises real-world policy success from 52.5% to 72.5% online and from 63.8% to 82.5% offline. These results establish temporal distance as a scalable supervision target and practical reward interface for generalist robot policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。