一个能看懂指令、图像和机器人状态的统一模型,让机器人自主完成复杂任务。
World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis

- 用自回归Transformer联合预测任务目标、视觉子目标和机器人动作
- 在真实世界任务中达到92.94%成功率,支持无标注视频直接学习新任务
- 推理仅需40毫秒,可动态启用世界预测以提升控制精度
我们提出世界-语言-动作(WLA)模型,一种新型具身基础模型。WLA以文本指令、图像和机器人状态为输入,联合预测文本子任务、子目标图像和机器人动作,融合了世界建模接口与语言推理能力。核心采用自回归Transformer架构,而非扩散模型,用于预测包含语义意图与精细物理动态的下一状态。物理动态由专用世界专家监督,并用于简化状态-动作关联建模。通过元查询使世界预测隐式影响动作生成,推理时可关闭世界预测;测试时可激活以实现性能扩展。原型WLA-0含20亿活跃参数,在NVIDIA RTX 5090上每步推理仅40毫秒。在仿真与真实环境评估中表现优异,如在RoboTwin2.0 Clean任务中达92.94%成功率,在RMBench任务中达56.5%成功率。该模型还具备从跨具身机器人视频中直接学习新任务的能力,无需动作标注。
原文摘要 · Abstract (English)
We propose world-language-action (WLA) models as a new class of embodied foundation models. WLA takes textual instructions, images, and robot states as inputs to jointly predict textual subtasks, subgoal images, and robot actions, conjoining the \emph{world modeling interface} to learn from extensive egocentric videos as in the world-action model (WAM) and the \emph{language reasoning} capacities to solve complex long-horizon tasks as in vision-language-action (VLA) models. At the core of WLA lies an \emph{autoregressive (AR)} Transformer backbone, instead of a bidirectional diffusion Transformer as in WAMs, to predict the \emph{next state}, comprising the \emph{semantic-level} textual intention and complementary \emph{fine-grained} physical dynamics. The physical dynamics are supervised by the world modeling objective based on a dedicated World Expert, and are leveraged to ease the characterization of the state-action correlation for the Action Expert. WLA leverages meta-queries to make the world prediction \emph{implicitly} impact the action generation so that the former can be disabled during inference. The world prediction can also be activated to enable test-time scaling for improved robot control. Our WLA-0 prototype, with 2B active parameters, achieves 40 ms per inference on an NVIDIA RTX 5090. Evaluations across simulated and real-world environments demonstrate that WLA-0 achieves state-of-the-art multi-task and long-horizon learning abilities, e.g., 92.94\% success rate on RoboTwin2.0 Clean and 56.5\% success rate on RMBench. WLA-0 also holds the promise to learn novel tasks directly from \emph{cross-embodiment robot videos} without action annotations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。