让智能体主动问问题,减少人类导航时的描述负担。
Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues
- 智能体通过自对话生成准确的观察描述,识别不确定目标。
- 仅在必要时向人提问,显著降低交互次数。
- 适合需要高效人机协作的机器人导航场景。
语言驱动的实例对象导航要求用户在导航前提供目标实例的详细描述,但这一过程对人类而言可能负担较重。为弥合这一差距,我们提出协同实例对象导航(CoIN)新任务设置:智能体在自然、无模板的开放对话中,于导航过程中主动解决对目标实例的不确定性。我们提出一种无需训练的新方法——带有不确定性感知的人机交互(AIUTA),该方法独立于导航策略,基于视觉-语言模型(VLMs)和大语言模型(LLMs)进行人机交互推理。首先,物体检测后,自提问模型在智能体内启动自对话,通过新颖的不确定性估计技术获取完整且准确的观察描述;随后,交互触发模块决定是否向人类提问、继续或终止导航,以最小化用户输入。为评估,我们构建了CoIN-Bench基准,包含专为复杂多实例场景设计的数据集,支持真人在线评估与模拟人机交互的可复现实验。在CoIN-Bench上,AIUTA表现优于现有语言驱动导航方法,尤其在复杂多实例场景中展现出更强鲁棒性。代码与基准将随论文录用公开于 https://intelligolabs.github.io/CoIN/
原文摘要 · Abstract (English)
Language-driven instance object navigation assumes that human users initiate the task by providing a detailed description of the target instance to the embodied agent. While this description is crucial for distinguishing the target from visually similar instances in a scene, providing it prior to navigation can be demanding for human. To bridge this gap, we introduce Collaborative Instance object Navigation (CoIN), a new task setting where the agent actively resolve uncertainties about the target instance during navigation in natural, template-free, open-ended dialogues with human. We propose a novel training-free method, Agent-user Interaction with UncerTainty Awareness (AIUTA), which operates independently from the navigation policy, and focuses on the human-agent interaction reasoning with Vision-Language Models (VLMs) and Large Language Models (LLMs). First, upon object detection, a Self-Questioner model initiates a self-dialogue within the agent to obtain a complete and accurate observation description with a novel uncertainty estimation technique. Then, an Interaction Trigger module determines whether to ask a question to the human, continue or halt navigation, minimizing user input. For evaluation, we introduce CoIN-Bench, with a curated dataset designed for challenging multi-instance scenarios. CoIN-Bench supports both online evaluation with humans and reproducible experiments with simulated user-agent interactions. On CoIN-Bench, we show that AIUTA serves as a competitive baseline, while existing language-driven instance navigation methods struggle in complex multi-instance scenes. Code and benchmark will be available upon acceptance at https://intelligolabs.github.io/CoIN/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。