DSWAM让机器人既能精细执行动作,又能理解语言分解复杂任务。
DSWAM: A Dual-System World Action Foundation Model for Fine-Grained Robot Manipulation

- 双系统设计:默认用视频世界模型直接执行动作,需分解时调用语言子任务规划器。
- 在真实机器人上实现90%以上任务成功率,比传统方法更稳定可靠。
- 适合需要复杂多步操作的家务机器人场景,如整理桌面或组装零件。
世界动作模型(WAMs)通过视频驱动的世界建模为机器人动作学习提供密集监督,优于依赖视觉-语言-动作(VLA)策略的现有方法。然而,现有WAMs缺乏语言级规划接口,难以将粗略指令分解为细粒度可执行子任务,尤其在涉及复杂多步目标的家庭任务中。同时,由于数据、机器人形态和任务协议差异,现有系统间缺乏公平的真实机器人对比。为此,本文提出双系统世界动作基础模型DSWAM,保留系统1的视频驱动执行路径,仅在需要时激活系统2的语言子任务规划器。该规划器基于短期视觉历史和全局任务提示预测可执行子任务;系统1则对每个指令或子任务进行基于世界感知的动作生成。执行器通过动作预测与视频联合训练,推理时直接输出动作块,无需显式生成未来视频。为实现在真实机器人上的高效部署,引入TensorRT加速、异步执行与实时分块(RTC)机制,避免策略查询阻塞控制。为实现与VLA政策的公平比较,所有实验均在匹配的机器人平台、预训练/后训练数据及评估标准下进行,采用DeMaVLA真实可变形操作设置。
原文摘要 · Abstract (English)
World Action Models (WAMs) provide a promising alternative to Vision-Language-Action (VLA) policies by using video-based world modeling as dense supervision for robot action learning. Existing WAMs excel at physically grounded execution, but typically lack the explicit language-level planning interface in VLM-based VLAs for decomposing coarse instructions. Such decomposition becomes important when household tasks involve complex multi-step goals, where coarse user commands need to be converted into sequences of fine-grained executable subtasks. Meanwhile, the field still lacks a fair real-robot comparison between VLA and WAM execution capabilities, since existing systems often differ in data, robot embodiments, and task protocols. To address both the decomposition gap and the need for a controlled WAM-VLA comparison, we introduce DSWAM, a Dual-System World Action Foundation Model for fine-grained robot manipulation. DSWAM keeps a System 1 WAM executor as the default control path and optionally activates a System 2 vision-language subtask planner only when task decomposition is useful. The planner predicts executable subtasks from short-term visual history and a global task prompt, while the WAM executor performs world-aware action generation for each instruction or subtask. The executor is trained with action prediction and video co-training, but inference directly predicts action chunks without explicit future video generation. To make this execution path practical on real robots, we further integrate TensorRT acceleration, asynchronous execution, and real-time chunking (RTC) so that policy queries do not block robot control. To provide a fair real-robot comparison with VLA policies, we build and evaluate DSWAM under the DeMaVLA real-world deformable manipulation setting with matched robot platform, pretraining data, post-training data, and evaluation criteria.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。