通过视觉多样性和区域感知提示学习,提升零样本人-物交互检测精度
Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection
- 引入群体视觉差异提示策略,增强对同一动词不同姿态的泛化能力
- 在四个零样本设置下于HICO-DET上达到最新最佳性能
- 适合关注零样本视觉理解与提示工程的研究者
零样本人-物交互检测旨在定位图像中的人与物体并识别其交互关系,即使特定动词-物体组合在训练中未见过。现有方法利用预训练视觉语言模型(如CLIP)进行提示学习,在共享嵌入空间中对齐自然语言提示与视觉特征,但仍难以应对交互的视觉复杂性:(1) 类内视觉多样性,即同一动词的不同实例呈现多种姿势和上下文;(2) 类间视觉混淆,不同动词产生视觉相似模式。为此,我们提出VDRP框架——视觉多样性与区域感知提示学习。首先,设计视觉多样性感知提示学习策略,将群体级视觉方差注入上下文嵌入,并通过高斯扰动促使提示捕捉动词的多样化视觉变化。其次,从人体、物体及二者联合区域中检索区域特异性概念,用于增强多样性感知提示嵌入,生成具有区域感知能力的提示,从而提升动词级别的判别力。在HICO-DET基准上的实验表明,该方法在四种零样本评估设置下均达到当前最优表现,有效缓解了类内多样性与类间混淆问题。代码已开源:https://github.com/mlvlab/VDRP。
原文摘要 · Abstract (English)
Zero-shot Human-Object Interaction detection aims to localize humans and objects in an image and recognize their interaction, even when specific verb-object pairs are unseen during training. Recent works have shown promising results using prompt learning with pretrained vision-language models such as CLIP, which align natural language prompts with visual features in a shared embedding space. However, existing approaches still fail to handle the visual complexity of interaction, including (1) intra-class visual diversity, where instances of the same verb appear in diverse poses and contexts, and (2) inter-class visual entanglement, where distinct verbs yield visually similar patterns. To address these challenges, we propose VDRP, a framework for Visual Diversity and Region-aware Prompt learning. First, we introduce a visual diversity-aware prompt learning strategy that injects group-wise visual variance into the context embedding. We further apply Gaussian perturbation to encourage the prompts to capture diverse visual variations of a verb. Second, we retrieve region-specific concepts from the human, object, and union regions. These are used to augment the diversity-aware prompt embeddings, yielding region-aware prompts that enhance verb-level discrimination. Experiments on the HICO-DET benchmark demonstrate that our method achieves state-of-the-art performance under four zero-shot evaluation settings, effectively addressing both intra-class diversity and inter-class visual entanglement. Code is available at https://github.com/mlvlab/VDRP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。