用视觉语言模型提升机器人对用户意图的理解与实时辅助能力
Utilizing Vision-Language Models as Action Models for Intent Recognition and Assistance
- 结合视觉语言模型与文本大模型构建语义先验，筛选相关目标
- 通过对象检测与实例分割，实现基于任务提示的上下文相关性评分
- 支持动态意图适应，适用于人机协作中的实时辅助场景
人机协作要求机器人快速推断用户意图，提供透明推理，并协助达成目标。我们此前提出GUIDER框架以推断导航与操作意图。本文提出在GUIDER中引入视觉语言模型（VLM）和纯文本语言模型（LLM），构建语义先验，根据任务提示过滤物体与位置。视觉管道（使用YOLO进行目标检测，Segment Anything Model进行实例分割）将候选物体裁片输入VLM，由其根据操作者提示评分相关性；同时，检测到的物体标签列表由纯文本LLM排序。这些评分加权于GUIDER原有的导航与操作层，选择上下文相关的目标并抑制无关对象。当联合信念超过阈值时，系统切换至自主模式，使机器人导航至目标区域并取回所需物品，同时适应操作者意图变化。未来工作将在Isaac Sim中使用Franka Emika机械臂搭载于Ridgeback底盘进行评估，重点关注实时辅助表现。
原文摘要 · Abstract (English)
Human-robot collaboration requires robots to quickly infer user intent, provide transparent reasoning, and assist users in achieving their goals. Our recent work introduced GUIDER, our framework for inferring navigation and manipulation intents. We propose augmenting GUIDER with a vision-language model (VLM) and a text-only language model (LLM) to form a semantic prior that filters objects and locations based on the mission prompt. A vision pipeline (YOLO for object detection and the Segment Anything Model for instance segmentation) feeds candidate object crops into the VLM, which scores their relevance given an operator prompt; in addition, the list of detected object labels is ranked by a text-only LLM. These scores weight the existing navigation and manipulation layers of GUIDER, selecting context-relevant targets while suppressing unrelated objects. Once the combined belief exceeds a threshold, autonomy changes occur, enabling the robot to navigate to the desired area and retrieve the desired object, while adapting to any changes in the operator's intent. Future work will evaluate the system on Isaac Sim using a Franka Emika arm on a Ridgeback base, with a focus on real-time assistance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。