arXiv:2506.21539cs.ROcs.AI2025-06被引 303

将动作与视觉理解生成统一建模,实现环境物理规律学习与动作优化的双向提升。

WorldVLA: Towards Autoregressive Action World Model

  • 构建统一框架,让世界模型和动作模型相互增强,共同预测未来图像并生成动作。
  • 在动作序列生成任务中,错误会随自回归过程累积,导致性能下降。
  • 提出注意力掩码策略,选择性屏蔽历史动作,显著提升连续动作生成效果。

我们提出 WorldVLA,一个统一动作与图像理解及生成的自回归动作世界模型。该模型将视觉-语言-动作(VLA)模型与世界模型融合于单一框架中:世界模型利用动作与图像理解预测未来图像,以学习环境底层物理规律,从而优化动作生成;动作模型则基于图像观测生成后续动作,辅助视觉理解,并反向促进世界模型的视觉生成能力。实验表明,WorldVLA 在性能上优于独立的世界模型与动作模型,凸显二者间的协同增益。然而,自回归方式生成动作序列时,动作模型表现下降,这归因于其对动作预测的泛化能力有限,导致早期动作错误被逐层传播。为此,我们设计一种注意力掩码策略,生成当前动作时有选择地屏蔽先前动作,显著提升了动作块生成任务的表现。

原文摘要 · Abstract (English)

We present WorldVLA, an autoregressive action world model that unifies action and image understanding and generation. Our WorldVLA intergrates Vision-Language-Action (VLA) model and world model in one single framework. The world model predicts future images by leveraging both action and image understanding, with the purpose of learning the underlying physics of the environment to improve action generation. Meanwhile, the action model generates the subsequent actions based on image observations, aiding in visual understanding and in turn helps visual generation of the world model. We demonstrate that WorldVLA outperforms standalone action and world models, highlighting the mutual enhancement between the world model and the action model. In addition, we find that the performance of the action model deteriorates when generating sequences of actions in an autoregressive manner. This phenomenon can be attributed to the model's limited generalization capability for action prediction, leading to the propagation of errors from earlier actions to subsequent ones. To address this issue, we propose an attention mask strategy that selectively masks prior actions during the generation of the current action, which shows significant performance improvement in the action chunk generation task.

世界模型动作生成自回归多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。