通过局部特征建模交互,提升零样本人物-物体关系检测性能
FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection

- 基于密集补丁级特征,显式建模人与物的局部空间关系
- 在未见交互上性能超越现有方法,尤其在零样本场景下优势显著
- 适合需要细粒度空间推理的视觉理解任务研究者
人-物交互(HOI)检测旨在定位图像中的人和物体并分类其交互行为。零样本HOI关注训练中未出现过的交互类型,要求模型具备跨组合泛化能力。现有方法虽利用视觉语言模型(VLMs)获得丰富语义表征,但多依赖全局或检测器中心特征,压缩了交互线索,限制了细粒度空间推理。为此,本文提出FineHOI框架,显式从密集补丁级特征中建模交互。受启发于交互由局部空间关系定义,我们引入自适应部件级注意力模块,通过无监督聚类将人与物分解为语义连贯部件,并根据交互相关性重加权。这些部件特征与全局特征融合后,经区域感知交互变换器生成最终的HOI嵌入。大量实验表明,FineHOI持续优于现有零样本HOI方法,在未见交互上表现尤为突出。代码已开源:https://github.com/francescotonini/fine-hoi。
原文摘要 · Abstract (English)
Human-Object Interaction (HOI) detection aims to localize humans and objects in images and classify their interactions. Zero-shot HOI focuses on recognizing interactions that are not observed during training, requiring models to generalize beyond seen verb-object compositions. Recent approaches leverage Vision-Language Models (VLMs), benefiting from rich semantic representations. However, they often rely on global or detector-centric features that compress interaction cues and hinder fine-grained spatial reasoning. To overcome this limitation, we propose FineHOI, a zero-shot HOI framework that explicitly models interactions from dense patch-level features. Our approach is motivated by the observation that human-object interactions are defined by localized spatial relationships, which are not preserved by global and detector-centric representations. To this end, we introduce an Adaptive Part-Level Attention module that decomposes humans and objects into semantically coherent parts via unsupervised clustering, and re-weights them based on their interaction relevance. These representations are then integrated through a Region-Aware Interaction Transformer that integrates part-aware and global features and produces the final HOI embedding. Extensive experiments demonstrate that FineHOI consistently outperforms existing zero-shot HOI methods, achieving particularly strong gains on unseen interactions. Code is available at https://github.com/francescotonini/fine-hoi.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。