用DINOv3模型实现高效开集人物交互检测,无需额外训练。
Streamlined Open-Vocabulary Human-Object Interaction Detection
- 仅用DINOv3模型,通过融合图像与查询特征桥接跨模态差距。
- 在SWiG-HOI和HICO-DET上达到当前最优性能。
- 参数冻结+少量可学习参数,适配快、适合快速部署场景。
开集人-物交互(HOI)检测旨在定位并识别图像中所有交互行为,包括训练中未见的类别。现有方法通常依赖传统HOI检测器与视觉语言模型(VLM)协同工作,但跨模型特征融合因表征差异大而困难。为此,我们提出SL-HOI——一种基于强大DINOv3模型的轻量化开集HOI检测框架。设计充分利用DINOv3组件的互补优势:其骨干网络实现细粒度定位,文本对齐视觉头支持开集交互分类。为促进交互查询与视觉头输出间的顺畅交叉注意力,首次将交互查询与骨干网络图像标记一同输入视觉头,有效弥合表征鸿沟。本方法中所有DINOv3参数均冻结,仅添加少量可学习参数,实现对HOI任务的快速适应。大量实验表明,SL-HOI在SWiG-HOI与HICO-DET基准上均取得当前最优表现,验证了架构的有效性。代码已开源。
原文摘要 · Abstract (English)
Open-vocabulary human-object interaction (HOI) detection aims to localize and recognize all human-object interactions in an image, including those unseen during training. Existing approaches usually rely on the collaboration between a conventional HOI detector and a Vision-Language Model (VLM) to recognize unseen HOI categories. However, feature fusion in this paradigm is challenging due to significant gaps in cross-model representations. To address this issue, we introduce SL-HOI, a StreamLined open-vocabulary HOI detection framework based solely on the powerful DINOv3 model. Our design leverages the complementary strengths of DINOv3's components: its backbone for fine-grained localization and its text-aligned vision head for open-vocabulary interaction classification. Moreover, to facilitate smooth cross-attention between the interaction queries and the vision head's output, we propose first feeding both the interaction queries and the backbone image tokens into the vision head, effectively bridging their representation gaps. All DINOv3 parameters in our approach are frozen, with only a small number of learnable parameters added, allowing a fast adaptation to the HOI detection task. Extensive experiments show that SL-HOI achieves state-of-the-art performance on both the SWiG-HOI and HICO-DET benchmarks, demonstrating the effectiveness of our streamlined model architecture. Code is available at https://github.com/MPI-Lab/SL-HOI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。