让机器人通过视觉注意力判断何时提问，更懂人类意图。
CLUE: Crossmodal disambiguation via Language-vision Understanding with attEntion
- 将视觉-语言模型的注意力转为显式空间信号，判断是否需要提问。
- 在真实交互数据上训练，比现有方法更准确地检测模糊性。
- 参数高效微调，适合资源受限的机器人交互场景。
随着机器人融入日常生活，人机交互日益复杂。交互式视觉定位（IVG）要求机器人理解人类意图并化解歧义。现有模型通常缺乏明确的提问决策机制，依赖隐式表示。CLUE通过将视觉-语言模型的跨模态注意力转化为显式的、空间定位的信号，实现主动提问判断。研究提取文本到图像的注意力图，输入轻量级CNN以检测指代歧义；同时使用LoRA微调的解码器进行对话并输出定位标记。模型在真实交互数据集和混合歧义数据集上训练，仅需InViG监督即超越当前最优方法，且参数高效。整体上，CLUE将模型内部注意力转化为可解释的提问决策信号。数据与代码公开于mouadabrini.github.io/clue。
原文摘要 · Abstract (English)
With the increasing integration of robots into daily life, human-robot interaction has become more complex and multifaceted. A critical component of this interaction is Interactive Visual Grounding (IVG), through which robots must interpret human intentions and resolve ambiguity. Existing IVG models generally lack a mechanism to determine when to ask clarification questions, as they implicitly rely on their learned representations. CLUE addresses this gap by converting the VLM's cross-modal attention into an explicit, spatially grounded signal for deciding when to ask. We extract text to image attention maps and pass them to a lightweight CNN to detect referential ambiguity, while a LoRA fine-tuned decoder conducts the dialog and emits grounding location tokens. We train on a real-world interactive dataset for IVG, and a mixed ambiguity set for the detector. With InViG-only supervision, our model surpasses a state-of-the-art method while using parameter-efficient fine-tuning. Similarly, the ambiguity detector outperforms prior baselines. Overall, CLUE turns the internal cross-modal attention of a VLM into an explicit, spatially grounded signal for deciding when to ask. The data and code are publicly available at: mouadabrini.github.io/clue
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。