用双路协作提升视觉语言模型在开放词汇人-物交互检测中的精度
Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection
- 通过注意力偏置引导视觉语言模型生成细粒度交互特征
- 在HICO-DET和V-COCO上实现开集与闭集任务的领先性能
- 适合研究开放词汇检测与多模态模型协同的学者参考
开放词汇人-物交互（HOI）检测旨在识别图像中所有感兴趣的 <人, 动词, 物体> 三元组，即使这些组合未在训练集中出现。现有方法通常依赖大视觉语言模型（VLM）输出的特征来增强交互表示的泛化能力，但这些视觉特征为整体性、粗粒度的，与检测任务需求相悖。为此，本文提出一种新型双路协作框架BC-HOI，包含两个核心组件：注意力偏置引导（ABG），利用HOI检测器提供的注意力偏置，指导VLM生成实例级细粒度交互特征；以及基于大语言模型（LLM）的监督引导（LSG），通过VLM中的LLM组件提供令牌级细粒度监督，强化ABG生成高质量注意力偏置的能力。在两个主流基准数据集HICO-DET和V-COCO上的大量实验表明，该方法在开放词汇和封闭设置下均取得优异表现。代码将开源于GitHub。
原文摘要 · Abstract (English)
Open vocabulary Human-Object Interaction (HOI) detection is a challenging task that detects all <human, verb, object> triplets of interest in an image, even those that are not pre-defined in the training set. Existing approaches typically rely on output features generated by large Vision-Language Models (VLMs) to enhance the generalization ability of interaction representations. However, the visual features produced by VLMs are holistic and coarse-grained, which contradicts the nature of detection tasks. To address this issue, we propose a novel Bilateral Collaboration framework for open vocabulary HOI detection (BC-HOI). This framework includes an Attention Bias Guidance (ABG) component, which guides the VLM to produce fine-grained instance-level interaction features according to the attention bias provided by the HOI detector. It also includes a Large Language Model (LLM)-based Supervision Guidance (LSG) component, which provides fine-grained token-level supervision for the HOI detector by the LLM component of the VLM. LSG enhances the ability of ABG to generate high-quality attention bias. We conduct extensive experiments on two popular benchmarks: HICO-DET and V-COCO, consistently achieving superior performance in the open vocabulary and closed settings. The code will be released in Github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。