让机器人同时理解语言、视觉并生成动作，提升任务完成率。
MM-ACT: Learn from Multimodal Parallel Generation to Act
- 统一建模文本、图像和动作，共享语义空间并并行生成。
- 在仿真与真实机器人上分别达到96.3%和72.0%的成功率。
- 通过跨模态学习增强动作生成，适合通用机器人控制场景。
通用机器人策略需要兼具语义理解能力与环境交互的预测能力。为此，我们提出MM-ACT，一种统一的视觉-语言-动作（VLA）模型，将文本、图像和动作映射到共享的标记空间，并实现三者间的联合生成。该模型采用重掩码并行解码策略生成文本与图像，以一步并行解码策略生成动作，提升效率。引入上下文共享多模态学习范式，在共享上下文中监督三模态生成，通过跨模态学习增强动作生成能力。在LIBERO仿真环境、Franka真实机器人及RoboTwin2.0平台上进行实验，评估域内与域外性能表现。结果表明，方法在LIBERO上取得96.3%的成功率，在真实Franka三个任务中平均达72.0%，在RoboTwin2.0八项双臂任务中为52.38%，且跨模态学习带来9.25%的性能增益。代码、模型与数据已开源。
原文摘要 · Abstract (English)
A generalist robotic policy needs both semantic understanding for task planning and the ability to interact with the environment through predictive capabilities. To tackle this, we present MM-ACT, a unified Vision-Language-Action (VLA) model that integrates text, image, and action in shared token space and performs generation across all three modalities. MM-ACT adopts a re-mask parallel decoding strategy for text and image generation, and employs a one-step parallel decoding strategy for action generation to improve efficiency. We introduce Context-Shared Multimodal Learning, a unified training paradigm that supervises generation in all three modalities from a shared context, enhancing action generation through cross-modal learning. Experiments were conducted on the LIBERO simulation and Franka real-robot setups as well as RoboTwin2.0 to assess in-domain and out-of-domain performances respectively. Our approach achieves a success rate of 96.3% on LIBERO, 72.0% across three tasks of real Franka, and 52.38% across eight bimanual tasks of RoboTwin2.0 with an additional gain of 9.25% from cross-modal learning. We release our codes, models and data at https://github.com/HHYHRHY/MM-ACT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。