arXiv:2411.08451cs.CV2024-11被引 2

让智能体更准理解手势指向,尤其在远距离时表现更优

AD-DINO: Attention-Dynamic DINO for Distance-Aware Embodied Reference Understanding

  • 引入动态注意力触线,根据交互距离调整手势表征
  • 在0.25 IoU下达76.4%准确率,0.75 IoU超越人类表现
  • 适合需要精准手势理解的机器人、人机交互场景

具身指代理解对智能体基于手势和语言描述预测目标至关重要。本文提出注意力动态DINO(AD-DINO),旨在减少不同交互情境下对指向手势的误判。该方法融合视觉与文本特征,同时预测目标物体的边界框及手势关注源。利用非语言交流中的距离感知特性,扩展虚拟触线机制,提出注意力动态触线,以交互距离为基础表征指向手势。该距离感知方法与独立预测关注源相结合,增强了物体与手势表征线之间的对齐。在YouRefIt数据集上的大量实验表明,该手势信息理解方法显著提升任务性能:在0.25 IoU阈值下达到76.4%准确率,并在0.75 IoU阈值下首次超越人类表现。与以往无距离感知方法的对比实验进一步验证了注意力动态触线在多种情境下的优越性。

原文摘要 · Abstract (English)

Embodied reference understanding is crucial for intelligent agents to predict referents based on human intention through gesture signals and language descriptions. This paper introduces the Attention-Dynamic DINO, a novel framework designed to mitigate misinterpretations of pointing gestures across various interaction contexts. Our approach integrates visual and textual features to simultaneously predict the target object's bounding box and the attention source in pointing gestures. Leveraging the distance-aware nature of nonverbal communication in visual perspective taking, we extend the virtual touch line mechanism and propose an attention-dynamic touch line to represent referring gesture based on interactive distances. The combination of this distance-aware approach and independent prediction of the attention source, enhances the alignment between objects and the gesture represented line. Extensive experiments on the YouRefIt dataset demonstrate the efficacy of our gesture information understanding method in significantly improving task performance. Our model achieves 76.4% accuracy at the 0.25 IoU threshold and, notably, surpasses human performance at the 0.75 IoU threshold, marking a first in this domain. Comparative experiments with distance-unaware understanding methods from previous research further validate the superiority of the Attention-Dynamic Touch Line across diverse contexts.

具身智能手势理解视觉推理距离感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。