用视觉语言模型实现带推理的智能规划，让机器像人一样思考下一步。
Planning with Reasoning using Vision Language World Model
- 基于视觉语言的世界模型，通过自洽推理生成动作与状态轨迹。
- 在人类评估中，高级规划系统使评分提升27%，优于现有方法。
- 适合研究具身智能、通用机器人与复杂任务规划的开发者。
高效规划依赖强大的世界模型，但能够理解并推理动作语义与时间抽象的高层世界模型仍不成熟。我们提出视觉语言世界模型（VLWM），一种基于自然视频训练的语言驱动世界建模基础模型。给定视觉输入，VLWM首先推断整体目标达成情况，再预测由交错的动作和世界状态变化组成的轨迹。这些目标通过基于树状摘要的压缩未来观察进行迭代式大模型自我修正来提取。VLWM同时学习动作策略与动态模型，分别支持反应式系统-1计划解码和通过成本最小化实现的反思式系统-2规划。成本由我们自监督训练的评判模型衡量，评估的是虚拟未来状态与预期目标状态之间的语义距离。在标准基准测试和我们提出的PlannerArena人类评估中，VLWM均达到视觉规划辅助任务（VPA）的领先性能，其中系统-2比系统-1提升27%的埃洛评分。在RoboVQA和WorldPrediction基准上，该模型也超越了强基线视觉语言模型。
原文摘要 · Abstract (English)
Effective planning requires strong world models, but high-level world models that can understand and reason about actions with semantic and temporal abstraction remain largely underdeveloped. We introduce the Vision Language World Model (VLWM), a foundation model trained for language-based world modeling on natural videos. Given visual observations, the VLWM first infers the overall goal achievements then predicts a trajectory composed of interleaved actions and world state changes. Those targets are extracted by iterative LLM Self-Refine conditioned on compressed future observations represented by Tree of Captions. The VLWM learns both an action policy and a dynamics model, which respectively facilitates reactive system-1 plan decoding and reflective system-2 planning via cost minimization. The cost evaluates the semantic distance between the hypothetical future states given by VLWM roll-outs and the expected goal state, and is measured by a critic model that we trained in a self-supervised manner. The VLWM achieves state-of-the-art Visual Planning for Assistance (VPA) performance on both benchmark evaluations and our proposed PlannerArena human evaluations, where system-2 improves the Elo score by +27% upon system-1. The VLWM models also outperforms strong VLM baselines on RoboVQA and WorldPrediction benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。