用视觉点指动作精准定位目标物，提升人机协作自然性
Pointing-Guided Target Estimation via Transformer-Based Attention
- 通过跨模态注意力机制融合2D指点手势与物体位置信息
- 在单目RGB数据下实现90%以上的目标预测准确率
- 适合需要自然交互的机器人场景，如桌面协作任务
指代性手势（如指向）是人类非语言交流的基本形式，有助于引导注意力至特定物体或位置。在人机交互中，机器人需能预测人类意图并做出恰当响应。本文提出多模态交互变换器（MM-ITF），一种模块化架构，用于在受控桌面场景中，基于NICOL机器人对人类自然指向动作进行目标预测。该模型利用跨模态注意力机制，将二维指向动作映射到物体位置，为每个候选物体分配置信度得分，并识别最可能的目标。实验表明，仅使用单目RGB图像，该方法即可准确预测目标物体，实现直观、易用的人机协作。为评估性能，我们引入补丁混淆矩阵，揭示模型在候选目标位置上的预测表现。代码已开源：https://github.com/lucamuellercode/MMITF。
原文摘要 · Abstract (English)
Deictic gestures, like pointing, are a fundamental form of non-verbal communication, enabling humans to direct attention to specific objects or locations. This capability is essential in Human-Robot Interaction (HRI), where robots should be able to predict human intent and anticipate appropriate responses. In this work, we propose the Multi-Modality Inter-TransFormer (MM-ITF), a modular architecture to predict objects in a controlled tabletop scenario with the NICOL robot, where humans indicate targets through natural pointing gestures. Leveraging inter-modality attention, MM-ITF maps 2D pointing gestures to object locations, assigns a likelihood score to each, and identifies the most likely target. Our results demonstrate that the method can accurately predict the intended object using monocular RGB data, thus enabling intuitive and accessible human-robot collaboration. To evaluate the performance, we introduce a patch confusion matrix, providing insights into the model's predictions across candidate object locations. Code available at: https://github.com/lucamuellercode/MMITF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。