arXiv:2602.15124cs.CV2026-02被引 1

用大模型实现无需训练的零样本人-物交互识别,通用性强且可适配任意检测器。

Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition

  • 将检测与交互识别解耦,用多模态大模型做零样本问答式交互识别
  • 在HICO-DET和V-COCO上达到领先零样本性能,跨数据集泛化能力强
  • 支持任意检测器接入,无需重训,适合需要快速部署的开放场景

零样本人-物交互(HOI)检测旨在定位图像中的人与物体并识别其交互关系。尽管开放词汇目标检测取得进展,但因交互组合多样性,交互识别仍具挑战。现有方法多为两阶段、依赖特定检测器,并使用粗粒度视觉-语言模型特征,限制了对未见交互的泛化能力。本文提出一种解耦框架,将目标检测与交互识别分离,利用多模态大模型(MLLM)实现零样本交互识别。设计确定性生成方法,将交互识别建模为视觉问答任务,强制确定性输出,实现无训练零样本识别。为进一步提升性能与效率,引入空间感知池化模块,融合外观与成对空间线索;设计单次确定性匹配方法,一次前向传播即可预测所有候选交互。在HICO-DET与V-COCO上的实验表明,该方法在零样本性能、跨数据集泛化能力方面均表现优异,且可无缝集成任意目标检测器而无需重新训练。代码已开源:https://github.com/SY-Xuan/DA-HOI。

原文摘要 · Abstract (English)

Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions. While advances in open-vocabulary object detection provide promising solutions for object localization, interaction recognition (IR) remains challenging due to the combinatorial diversity of interactions. Existing methods, including two-stage methods, tightly couple IR with a specific detector and rely on coarse-grained vision-language model (VLM) features, which limit generalization to unseen interactions. In this work, we propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR. We introduce a deterministic generation method that formulates IR as a visual question answering task and enforces deterministic outputs, enabling training-free zero-shot IR. To further enhance performance and efficiency by fine-tuning the model, we design a spatial-aware pooling module that integrates appearance and pairwise spatial cues, and a one-pass deterministic matching method that predicts all candidate interactions in a single forward pass. Extensive experiments on HICO-DET and V-COCO demonstrate that our method achieves superior zero-shot performance, strong cross-dataset generalization, and the flexibility to integrate with any object detectors without retraining. The codes are publicly available at https://github.com/SY-Xuan/DA-HOI.

零样本检测多模态大模型交互识别解耦框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。