arXiv:2510.03182cs.ROcs.AI2025-10被引 3

用双视觉语言模型自动生成规划规则,让AI能看图做复杂长程任务。

Simulation to Rules: A Dual-VLM Framework for Formal Visual Planning

  • 用模拟与生成双模型协同,自动生成可执行的视觉规划代码。
  • 在未见场景中达到70%规划成功率,未见外观下仍保持54.1%成功率。
  • 适用于多机器人协作、部分可观测等复杂3D任务,泛化能力强。

视觉语言模型(VLMs)在视觉规划中潜力巨大,但在空间精确性和长程推理上表现不足;而规划领域定义语言(PDDL)规划器虽擅长形式化长程规划,却无法处理视觉输入。现有方法将视觉问题转为PDDL以融合二者优势,但生成包含规划规则的PDDL域文件仍需人工干预或环境交互。本文提出VLMFP框架,通过一个模拟模型(SimVLM)预测动作后果,结合生成模型(GenVLM)迭代优化并校准符号执行结果,实现对未见实例、视觉外观和游戏规则的多层级泛化。在6个网格世界任务中,SimVLM在已见与未见外观下的场景理解与动作模拟准确率分别为87.3%和86.0%。借助其引导,VLMFP在已见外观下规划成功率达70.0%,未见外观下为54.1%。进一步验证显示,该框架可扩展至复杂3D长程任务,包括多机器人协作与装配场景,具备部分可观测性及多样化视觉变化适应能力。

原文摘要 · Abstract (English)

Vision Language Models (VLMs) show strong potential for visual planning but struggle with precise spatial and long-horizon reasoning, while Planning Domain Definition Language (PDDL) planners excel at formal long-horizon planning but cannot interpret visual inputs. Recent works combine these complementary advantages by translating visual problems into PDDL. However, while VLMs can generate PDDL problem files satisfactorily, accurately generating PDDL domain files, which encode planning rules, remains challenging and typically requires human expertise or environment interaction. We propose VLMFP, a Dual-VLM-guided framework that autonomously generates both PDDL problem and domain files for formal visual planning. VLMFP combines a SimVLM that simulates action consequences with a GenVLM that generates and iteratively refines PDDL files by aligning symbolic execution with simulated outcomes, enabling multiple levels of generalization across unseen instances, visual appearances, and game rules. We evaluate VLMFP on 6 grid-world domains and demonstrate its generalization capability. On average, SimVLM achieves 87.3% and 86.0% scenario understanding and action simulation for seen and unseen appearances, respectively. With the guidance of SimVLM, VLMFP attains 70.0%, 54.1% planning success on unseen instances in seen and unseen appearances, respectively. We further demonstrate that VLMFP scales to complex long-horizon 3D planning tasks, including multi-robot collaboration and assembly scenarios with partial observability and diverse visual variations. Project page: https://sites.google.com/view/vlmfp.

视觉规划PDDL多智能体泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。