让智能体按需切换文本与视觉推理，提升复杂任务表现
Dreaming when Necessary: Advancing World Action Models with Adaptive Multi-Modal Reasoning
- 根据任务阶段自动选择文本或视觉推理模式
- 在模拟与真实任务中均显著优于当前最优策略
- 适合需要长时序决策与多模态理解的机器人应用
世界动作模型（WAMs）为具身智能提供了有前景的解决方案，但现有方法过度依赖视频预测作为动作先验，缺乏自适应的多模态推理能力，限制了其在长时序、复杂任务中的表现。我们观察到，不同执行情境下WAMs需要不同的多模态推理模式：任务转换阶段需依赖文本推理以指导高层动作预测，精细操作阶段则需视觉推理实现精确控制。基于此，我们提出AdaWAM，一种具备自适应多模态推理能力的世界动作模型。AdaWAM引入轻量级动态路由机制，在任务执行过程中自主触发文本或视觉推理。在模拟与真实具身任务上的实验表明，AdaWAM在大幅提升推理效率的同时，显著超越现有先进具身策略。代码与演示见：https://adawam.github.io/
原文摘要 · Abstract (English)
World Action Models (WAMs) offer a promising approach to embodied intelligence, yet existing methods rely heavily on video prediction as action priors and lack adaptive multimodal reasoning, limiting their effectiveness on long-horizon, complex tasks. We observe that WAMs require different multimodal reasoning modes under different execution contexts: textual reasoning is essential during task transitions to guide high-level action prediction, while visual reasoning is critical during fine-grained manipulation for precise control. Motivated by this observation, we propose \textbf{AdaWAM}, a world action model with adaptive multimodal reasoning abilities. AdaWAM integrates a lightweight dynamic router that autonomously triggers textual or visual reasoning as needed during task execution. Experiments on both simulated and real-world embodied tasks show that AdaWAM substantially improves inference efficiency while outperforming state-of-the-art embodied policies. Codes and demos are available at: https://adawam.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。