用时间优先的统一标记建模机器人模仿学习，提升多源感知与动作生成的效率。
WorldToken: Time-First Sequence Modeling for Robotic Imitation Learning
- 将视觉、本体感觉和任务条件融合为每步一个世界标记，用因果Transformer建模序列。
- 在23个RoboCasa任务上实现59.45%平均闭环成功率，每任务仅需2,900条演示数据。
- 验证了时间上下文的重要性：减少历史输入会显著降低性能，证明时序建模关键性。
机器人策略在每个决策步接收异构观测，但现有序列模型对时间结构的处理方式各异。本文提出WorldToken，一种时间优先的策略实例化方法，将多视角图像、本体感觉和任务条件在每个策略时间步内融合为单一世界标记。通过因果时序Transformer建模该世界标记序列，并采用扩散动作头生成动作块。在23个RoboCasa任务上，一个85.3M参数的策略（除冻结预训练CLIP文本编码器外从零训练）在每任务仅2,900条生成演示数据下达到59.45%的平均闭环成功率。全面的五种数据集规模、五种模型规模及两种训练种子的因子实验表明，目标域数据越多性能越好，但模型规模超过中等水平后收益递减。在相同检查点历史截断条件下，将可见历史缩减至1或2个策略时间步，导致全部50个RoboCasa策略的闭环成功率下降。在RMBench Blocks Ranking测试中，将可见历史从146秒降至8秒使评估器成功率从95%降至28%，而探索性长回放可维持参考交换序列超过850秒。这些结果证实了完整WorldToken实例化的可行性，并刻画了其在测试设置下的数据扩展与时间上下文行为。但未证明其相对于其他序列组织方式的优越性，也未分离出具体组件对性能的贡献。
原文摘要 · Abstract (English)
Robot policies receive heterogeneous observations at each decision step, yet sequence models differ in how they organize these inputs over time. We introduce WorldToken, a time-first policy instantiation that fuses multiview images, proprioception, and task conditioning within each policy timestep into one world token. A causal temporal Transformer models the resulting world-token sequence, and a diffusion action head generates action chunks. On 23 RoboCasa tasks, an 85.3M-parameter policy trained from scratch apart from a frozen pretrained CLIP text encoder achieves 59.45% mean closed-loop success using 2,900 generated demonstrations per task. A complete factorial sweep over five dataset sizes, five model sizes, and two training seeds shows consistent gains from additional target-domain data and diminishing returns beyond moderate model size. Under same-checkpoint history truncation, reducing visible history to one or two policy timesteps lowers closed-loop success for all 50 RoboCasa policies. On RMBench Blocks Ranking, reducing visible history from 146 to 8 seconds lowers evaluator success from 95% to 28%, while an exploratory extended rollout sustains the reference swap sequence for over 850 seconds. These results establish the empirical feasibility of the complete WorldToken instantiation and characterize its data-scaling and temporal-context behavior under the tested recipes. They do not establish superiority over alternative sequence organizations or isolate which components of the complete implementation drive the observed performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。