让视觉语言模型把任务转成可验证的规划语言,提升长程决策能力。
Vision Language Models Cannot Plan, but Can They Formalize?
- 用VLM将多模态任务转化为PDDL格式,交由形式化求解器生成计划。
- 在真实多视角低质量图像上,性能远超端到端生成,强模型已克服物体关系识别瓶颈。
- 适合研究多模态规划、形式化推理与视觉语言模型应用的学者参考。
视觉语言模型(VLMs)虽能完成简单多模态规划任务,但难以应对需长时间序列动作的复杂任务。在纯文本环境中,通过将大语言模型(LLMs)角色从直接生成动作序列转为将规划问题转化为形式化语言(如PDDL),显著提升了长程规划能力。然而,在多模态场景中,现有研究仍匮乏,常依赖预定义物体词汇或高度相似的少样本示例。本文提出五种VLM-as-formalizer流水线,支持单次输入、开放词汇和多模态的PDDL形式化。我们在现有基准上评估,并引入两个新基准,首次涵盖真实多视角、低质量图像下的规划任务。结果表明,该方法显著优于端到端规划生成;弱VLM的主要瓶颈在于物体关系的视觉定位,而强模型已基本克服此问题。尽管中间文本表示(如描述或场景图)部分缓解性能损失,但其收益不一致,仍留有改进空间。
原文摘要 · Abstract (English)
The advancement of vision language models (VLMs) has empowered embodied agents to accomplish simple multimodal planning tasks, but not long-horizon ones requiring long sequences of actions. In text-only simulations, long-horizon planning has seen significant improvement brought by repositioning the role of LLMs. Instead of directly generating action sequences, LLMs translate the planning domain and problem into a formal planning language like the Planning Domain Definition Language (PDDL), which can call a formal solver to derive the plan in a verifiable manner. In multimodal environments, research on VLM-as-formalizer remains scarce, usually involving gross simplifications such as predefined object vocabulary or overly similar few-shot examples. In this work, we present a suite of five VLM-as-formalizer pipelines that tackle one-shot, open-vocabulary, and multimodal PDDL formalization. We evaluate those on an existing benchmark while presenting another two that for the first time account for planning with authentic, multi-view, and low-quality images. We conclude that VLM-as-formalizer greatly outperforms end-to-end plan generation. We find that visual grounding of object relations remains the primary bottleneck for weaker VLMs, while stronger models have largely overcome this limitation. While generating intermediate, textual representations such as captions or scene graphs partially compensate for the performance, their inconsistent gain leaves headroom for future research directions on multimodal planning formalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。