利用大模型的词级语义空间信号,零样本实现动作相关区域精确定位。
Token-Based Affordance Grounding with Large Vision-Language Models

- 通过分析大视觉语言模型输出词元的空间注意力,筛选聚焦目标物体的词元。
- 在AGD20K未见划分上提升NSS 10.7%,HICO-IIF上提升29.7%。
- 无需额外标注,适合需要零样本动作定位的机器人与具身智能场景。
动作可及性定位旨在识别支持特定动作的图像区域,是物理智能与具身感知的核心能力。以往研究主要依赖外视角图像中的动作标签进行弱监督学习,但此类方法在包含共现动作的视觉模糊图像中表现不佳,且难以区分语义相似的动作,因现有方法通常依赖缺乏丰富语义细节的简短动作短语。尽管大视觉语言模型(LVLMs)编码了丰富的动作语义,其条件文本输出隐含空间线索,却无法直接提供动作特定的空间定位。为此,我们提出TokAG,一种利用LVLM词级语义-空间信号的零样本动作可及性定位框架。观察到不同输出词元关联的注意力图差异显著,许多关注背景等无关区域。因此,我们引入空间感知词元选择机制,系统评估每个输出词元,选取其注意力图在目标物体上激活占主导的词元,而非依赖任意注意力图。通过提取这些聚焦物体的注意力图,我们将LVLM的隐式语义信号转化为零样本动作可及性热图。所提零样本框架在多个基准上持续优于先前弱监督方法,在未见划分的AGD20K上提升NSS 10.7%,在HICO-IIF上提升29.7%。代码与模型将公开发布。
原文摘要 · Abstract (English)
Affordance grounding aims to localize image regions that support a specific action, serving as a core capability for physical intelligence and embodied perception. Previous studies have primarily relied on weakly supervised learning with action labels from exocentric images. However, these methods often struggle with visually ambiguous exocentric images containing co-occurring actions; moreover, they fail to distinguish semantically similar actions because existing methods typically rely on brief action phrases that lack rich semantic details for action-specific localization. Although large vision-language models (LVLMs) encode rich action semantics and their action-conditioned textual outputs implicitly contain spatial cues, they do not directly provide action-specific spatial localization. To address these problems, we propose TokAG, a zero-shot affordance grounding framework that exploits the token-level semantic-spatial signals in LVLMs to localize action-relevant regions without external supervision. We observe that attention maps associated with different LVLM output tokens vary significantly, with many attending to irrelevant regions such as the background. Thus, we introduce a spatial-aware token-selection mechanism to systematically evaluate each output token and select the one whose attention maps exhibit dominant activation over the target object, instead of relying on arbitrary attention maps. By extracting these object-focused attention maps, we transform the LVLM's implicit semantic signals into zero-shot affordance heatmaps. Our zero-shot framework consistently outperforms prior weakly supervised approaches across multiple benchmarks, improving NSS by 10.7% on the unseen split of AGD20K and by 29.7% on HICO-IIF. The code and models will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。