用语言和视觉信息引导查询初始化,提升人物交互检测准确率
QueryCraft: Transformer-Guided Query Initialization for Enhanced Human-Object Interaction Detection
- 通过跨模态Transformer融合图像与文本,生成语义明确的查询
- 在HICO-Det和V-COCO上达到当前最佳性能,精度显著提升
- 适合做视觉理解、交互检测的研究者和开发者参考
人-物交互(HOI)检测旨在定位图像中的人与物体对并识别其交互动作。尽管基于DETR的方法已成为主流框架,但仍受限于随机初始化查询缺乏语义信息,导致检测效果不佳。为此,我们提出QueryCraft,一种可即插即用的HOI检测框架,通过基于Transformer的查询初始化引入语义先验与引导特征学习。核心是ACTOR(Action-aware Cross-modal Transformer),一个联合关注视觉区域与文本提示的跨模态Transformer编码器,用于提取与动作相关的特征。ACTOR不只对齐模态,更利用语言引导注意力推断交互语义,生成具有语义意义的查询表示。为进一步提升物体级查询质量,我们设计了感知蒸馏查询解码器(PDQD),从预训练检测器中蒸馏物体类别知识作为查询初始化。双分支查询初始化使模型能生成更可解释、高效的查询,显著提升检测性能。在HICO-Det与V-COCO基准上的大量实验表明,本方法达到当前最优表现,并具备强泛化能力。代码将在发表后开源。
原文摘要 · Abstract (English)
Human-Object Interaction (HOI) detection aims to localize human-object pairs and recognize their interactions in images. Although DETR-based methods have recently emerged as the mainstream framework for HOI detection, they still suffer from a key limitation: Randomly initialized queries lack explicit semantics, leading to suboptimal detection performance. To address this challenge, we propose QueryCraft, a novel plug-and-play HOI detection framework that incorporates semantic priors and guided feature learning through transformer-based query initialization. Central to our approach is \textbf{ACTOR} (\textbf{A}ction-aware \textbf{C}ross-modal \textbf{T}ransf\textbf{OR}mer), a cross-modal Transformer encoder that jointly attends to visual regions and textual prompts to extract action-relevant features. Rather than merely aligning modalities, ACTOR leverages language-guided attention to infer interaction semantics and produce semantically meaningful query representations. To further enhance object-level query quality, we introduce a \textbf{P}erceptual \textbf{D}istilled \textbf{Q}uery \textbf{D}ecoder (\textbf{PDQD}), which distills object category awareness from a pre-trained detector to serve as object query initiation. This dual-branch query initialization enables the model to generate more interpretable and effective queries for HOI detection. Extensive experiments on HICO-Det and V-COCO benchmarks demonstrate that our method achieves state-of-the-art performance and strong generalization. Code will be released upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。