arXiv:2602.01067cs.RO2026-02被引 16

用多模态数据联合训练,让机器人模型更懂指令、适应新任务。

A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulation

  • 融合视觉语言与跨机器人数据提升泛化能力
  • 5000小时数据训练下,新任务适应速度显著加快
  • 适合构建通用机器人操作系统的研究者参考

大行为模型通过在多任务机器人数据上大规模训练,展现出强大的灵巧操作能力,但其泛化性能受限于机器人数据覆盖不足。为在不增加昂贵数据采集的前提下扩展数据覆盖范围,近期工作采用联合训练策略:同时利用目标机器人数据和异构数据模态。然而,不同数据模态及训练策略对策略性能的影响尚不明确。本文开展大规模实证研究,考察五种联合训练数据模态:标准视觉-语言数据、机器人轨迹密集语言标注、跨体感机器人数据、人类操作视频、离散机器人动作标记,并对比单阶段与多阶段训练策略。研究基于4,000小时机器人与人类操作数据及5000万条视觉-语言样本,训练了视觉-语言-动作策略。在58,000次仿真滚动和2,835次真实世界滚动中评估了89个策略。结果表明,结合视觉-语言与跨体感机器人数据能显著提升对分布偏移、未见任务和语言指令的泛化能力;而离散动作标记未带来显著收益。有效模态组合产生累积增益,支持通过微调快速适应未见过的长时序灵巧任务。仅使用机器人数据训练会削弱视觉-语言模型主干的语义理解能力,而联合训练可恢复该能力。在仿真基准中,显式基于联合训练数据学习的思维链轨迹进行动作生成并未提升性能。整体结果为构建可扩展的通用机器人策略提供了实用指导。

原文摘要 · Abstract (English)

Large behavior models have shown strong dexterous manipulation capabilities by extending imitation learning to large-scale training on multi-task robot data, yet their generalization remains limited by the insufficient robot data coverage. To expand this coverage without costly additional data collection, recent work relies on co-training: jointly learning from target robot data and heterogeneous data modalities. However, how different co-training data modalities and strategies affect policy performance remains poorly understood. We present a large-scale empirical study examining five co-training data modalities: standard vision-language data, dense language annotations for robot trajectories, cross-embodiment robot data, human videos, and discrete robot action tokens across single- and multi-phase training strategies. Our study leverages 4,000 hours of robot and human manipulation data and 50M vision-language samples to train vision-language-action policies. We evaluate 89 policies over 58,000 simulation rollouts and 2,835 real-world rollouts. Our results show that co-training with forms of vision-language and cross-embodiment robot data substantially improves generalization to distribution shifts, unseen tasks, and language following, while discrete action token variants yield no significant benefits. Combining effective modalities produces cumulative gains and enables rapid adaptation to unseen long-horizon dexterous tasks via fine-tuning. Training exclusively on robot data degrades the visiolinguistic understanding of the vision-language model backbone, while co-training with effective modalities restores these capabilities. Explicitly conditioning action generation on chain-of-thought traces learned from co-training data does not improve performance in our simulation benchmark. Together, these results provide practical guidance for building scalable generalist robot policies.

机器人操作联合训练多模态学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。