用视觉语言模型让机器人根据环境自动规划带视觉判断的任务流程
VLM-driven Behavior Tree for Context-aware Task Planning
- 用VLM生成带自由文本条件的行动树，实现视觉感知驱动的任务编排
- 在真实咖啡厅场景中验证，能根据图像实时判断条件是否满足
- 适合需要环境感知的复杂服务机器人任务规划
大型语言模型（LLMs）用于生成行为树（BTs）在机器人领域受到关注，但尚处于初期阶段。本文提出一种新框架，利用视觉语言模型（VLMs）交互式生成和编辑行为树，以应对视觉条件，实现在视觉复杂环境中的上下文感知机器人操作。本方法的关键在于通过自提示视觉条件进行条件控制：VLM生成包含自由文本形式视觉条件节点的行为树；另一VLM过程将文本融入提示，在机器人执行时与真实图像进行条件评估。我们在真实咖啡厅场景中验证了该框架，展示了其可行性与局限性。
原文摘要 · Abstract (English)
The use of Large Language Models (LLMs) for generating Behavior Trees (BTs) has recently gained attention in the robotics community, yet remains in its early stages of development. In this paper, we propose a novel framework that leverages Vision-Language Models (VLMs) to interactively generate and edit BTs that address visual conditions, enabling context-aware robot operations in visually complex environments. A key feature of our approach lies in the conditional control through self-prompted visual conditions. Specifically, the VLM generates BTs with visual condition nodes, where conditions are expressed as free-form text. Another VLM process integrates the text into its prompt and evaluates the conditions against real-world images during robot execution. We validated our framework in a real-world cafe scenario, demonstrating both its feasibility and limitations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。