用视觉语言模型实现无需手写规则的通用任务规划,跨领域表现更优。
SYMBOLIZER: Symbolic Model-free Task Planning with VLMs

- 仅需描述物体关系的抽象谓词,由视觉语言模型从图像中解析状态。
- 不依赖动作模型,用通用启发式搜索在复杂状态空间中高效求解。
- 适用于新场景和未见过的任务,适合需要快速适配的机器人应用。
传统任务与运动规划(TAMP)系统依赖物理模型进行运动规划,以及手工构建的符号模型进行任务规划。尽管物理模型常可用,但符号模型(包含状态解释和动作模型)需人工精心设计或从标注数据中学习,过程耗时且限制于特定领域,难以扩展。相比之下,视觉语言模型(VLMs)具备出色的零样本视觉理解能力,但在规划方面仍表现有限。因此,将VLM与经典规划结合以实现长时程推理具有巨大潜力。现有方法仍缺乏通用性,依赖预定义对象或符号动作模型。本文提出一种可泛化至未见问题实例的框架:仅需描述物体间关系的抽象谓词,通过VLM从图像中对齐并生成符号状态;采用领域无关的启发式搜索(基于目标计数与宽度),无需动作模型。在VLM引导的状态空间上进行符号搜索,性能优于直接使用VLM的规划方法,并达到使用VLM导出启发式的现有方法水平。这表明,领域无关搜索可在大规模组合状态空间中有效求解跨领域问题。我们在ProDG和ViPlan基准上进行了广泛评估,结果达到当前最优水平。
原文摘要 · Abstract (English)
Traditional Task and Motion Planning (TAMP) systems depend on physics models for motion planning and discrete symbolic models for task planning. Although physics model are often available, symbolic models (consisting of symbolic state interpretation and action models) must be meticulously handcrafted or learned from labeled data. This process is both resource-intensive and constrains the solution to the specific domain, limiting scalability and adaptability. On the other hand, Visual Language Models (VLMs) show desirable zero-shot visual understanding (due to their extensive training on heterogeneous data), but still achieve limited planning capabilities. Therefore, integrating VLMs with classical planning for long-horizon reasoning in TAMP problems offers high potential. Recent works in this direction still lack generality and depend on handcrafted, task-specific solutions, e.g. describing all possible objects in advance, or using symbolic action models. We propose a framework that generalizes well to unseen problem instances. The method requires only lifted predicates describing relations among objects and uses VLMs to ground them from images to obtain the symbolic state. Planning is performed with domain-independent heuristic search using goal-count and width-based heuristics, without need for action models. Symbolic search over VLM-grounded state-space outperforms direct VLM-based planning and performs on par with approaches that use a VLM-derived heuristic. This shows that domain-independent search can effectively solve problems across domains with large combinatorial state spaces. We extensively evaluate on extensively evaluate our method and achieve state-of-the-art results on the ProDG and ViPlan benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。