通过动态提示与概念校准,提升开放词汇人-物交互检测性能。
Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration
- 设计交互感知提示生成器,动态生成聚焦关键模式的紧凑提示。
- 在SWIG-HOI和HICO-DET上显著超越现有方法,实现更优检测精度。
- 适合关注开放词汇视觉理解与细粒度关系建模的研究者使用。
开放词汇人-物交互(HOI)检测旨在识别人类与物体之间的交互,并泛化到训练集外的新交互类别。当前方法依赖视觉语言模型(VLMs),但受限于图像编码器的不足——图像级预训练难以对齐细粒度区域级交互检测需求;同时,文本描述的视觉外观编码仍具挑战,制约模型捕捉复杂交互关系的能力。为此,本文提出交互感知提示与概念校准(INP-CC),一种端到端的开放词汇HOI检测框架。具体而言,设计交互感知提示生成器,根据输入场景动态生成一组紧凑提示,实现相似交互间的注意力选择性共享,引导模型聚焦关键交互模式而非通用图像语义。此外,通过语言模型引导的概念校准,优化HOI概念表示,借助跨类别视觉相似性分析区分多样交互。还引入负样本策略,增强跨模态相似性建模能力,使模型更好区分视觉相似但语义不同的动作。大量实验表明,INP-CC在SWIG-HOI和HICO-DET数据集上显著优于现有先进模型。代码已开源:https://github.com/ltttpku/INP-CC。
原文摘要 · Abstract (English)
Open Vocabulary Human-Object Interaction (HOI) detection aims to detect interactions between humans and objects while generalizing to novel interaction classes beyond the training set. Current methods often rely on Vision and Language Models (VLMs) but face challenges due to suboptimal image encoders, as image-level pre-training does not align well with the fine-grained region-level interaction detection required for HOI. Additionally, effectively encoding textual descriptions of visual appearances remains difficult, limiting the model's ability to capture detailed HOI relationships. To address these issues, we propose INteraction-aware Prompting with Concept Calibration (INP-CC), an end-to-end open-vocabulary HOI detector that integrates interaction-aware prompts and concept calibration. Specifically, we propose an interaction-aware prompt generator that dynamically generates a compact set of prompts based on the input scene, enabling selective sharing among similar interactions. This approach directs the model's attention to key interaction patterns rather than generic image-level semantics, enhancing HOI detection. Furthermore, we refine HOI concept representations through language model-guided calibration, which helps distinguish diverse HOI concepts by investigating visual similarities across categories. A negative sampling strategy is also employed to improve inter-modal similarity modeling, enabling the model to better differentiate visually similar but semantically distinct actions. Extensive experimental results demonstrate that INP-CC significantly outperforms state-of-the-art models on the SWIG-HOI and HICO-DET datasets. Code is available at https://github.com/ltttpku/INP-CC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。