让视觉语言模型学会一步步纠错规划，提升真实场景决策能力。
CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates
- 用增量更新场景图实现视觉动作的逐步推理。
- 模型在新基准上平均提升4.4%，跨任务泛化表现更好。
- 适合研究视觉规划、具身智能与多模态推理的学者。
视觉语言模型在文本规划任务中表现良好，但在视觉决策方面仍显不足。为此，我们提出纠正性序列规划（CoSPlan）基准，要求模型从初始场景出发，通过一系列视觉动作到达目标场景。该基准评估模型生成连贯视觉步骤的能力（步骤完成度），并通过引入错误动作来检验其错误检测与修正能力，以促进对任务的深层理解。CoSPlan涵盖四类任务：迷宫导航、积木重排、图像重建和物体重组。尽管采用链式思维与场景图等先进推理策略，现有模型在该任务上仍表现不佳。为此，我们提出无需训练的场景图增量更新（SGI）方法，将图像转化为可迭代优化的‘文本化’场景图，支持逐步推理。SGI在CoSPlan上带来约4.4%的平均性能提升，并在PlanBench和VQA上展现出良好泛化能力。项目页面提供解谜工具链接。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) have shown promising planning capabilities, yet their success remains confined to the text domain, leaving visual decision-making relatively underexplored. Addressing this gap, we introduce Corrective Sequence Planning (CoSPlan) benchmark, where VLMs must plan a sequence of visual actions from an initial scene to a target scene. CoSPlan evaluates models on their ability to imagine and execute a coherent set of visual steps required to reach the goal (Step Completion). To prevent any shortcuts that simply describe the final scene, we introduce an erroneous action in decision-making, which must be detected (Error Detection) and corrected to reach the goal, enabling a deeper understanding of the task. CoSPlan spans across 4 tasks: maze navigation, block re-arrangement, image reconstruction, and object re-organization. Despite using advanced reasoning strategies such as Chain-of-Thought and Scene Graphs, VLMs struggle on CoSPlan, while still showing promising performance in the text domain. Addressing this, we propose Scene Graph Incremental updates (SGI), a novel training-free method to transform images into `textual' scene graphs, enabling step-by-step reasoning through iterative scene graph refinement. SGI yields an average of ~4.4% improvement on CoSPlan w/ generalization on PlanBench and VQA. Link for solving puzzles on the project page.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。