用多模态问答提升开放词汇目标检测,无需重训练即可增强模型表现。
MQADet: A Plug-and-Play Paradigm for Enhancing Open-Vocabulary Object Detection via Multimodal Question Answering
- 通过三阶段多模态问答引导大模型精准定位复杂目标
- 在四个数据集上显著提升未见类别检测性能,尤其在复杂场景中
- 即插即用设计,适配现有检测器,零成本扩展能力
开放词汇目标检测(OVD)旨在识别训练中未见过的任意类别物体,面临视觉-文本错位和长尾类别不平衡等挑战。为解决这些问题,我们提出MQADet,一种利用多模态大语言模型(MLLMs)跨模态推理能力的通用增强范式。该方法作为即插即用模块,可无缝集成到预训练检测器中,无需额外训练。我们设计了新颖的三阶段多模态问答(MQA)流程,引导MLLM精准定位复杂文本与视觉目标,同时增强检测器对相关对象的关注。我们在四个具有挑战性的开放词汇数据集上构建新基准,以三种先进检测器为基线进行评估。实验表明,该范式显著提升现有检测器性能,尤其在未见复杂类别上表现突出。代码将公开以促进后续研究。
原文摘要 · Abstract (English)
Open-vocabulary detection (OVD) is a challenging task to detect and classify objects from an unrestricted set of categories, including those unseen during training. Existing open-vocabulary detectors are limited by complex visual-textual misalignment and long-tailed category imbalances, leading to suboptimal performance in challenging scenarios. To address these limitations, we introduce MQADet, a universal paradigm for enhancing existing open-vocabulary detectors by leveraging the cross-modal reasoning capabilities of multimodal large language models (MLLMs). MQADet functions as a plug-and-play solution that integrates seamlessly with pre-trained object detectors without substantial additional training costs. Specifically, we design a novel three-stage Multimodal Question Answering (MQA) pipeline to guide the MLLMs to precisely localize complex textual and visual targets while effectively enhancing the focus of existing object detectors on relevant objects. To validate our approach, we present a new benchmark for evaluating our paradigm on four challenging open-vocabulary datasets, employing three state-of-the-art object detectors as baselines. Experimental results demonstrate that our proposed paradigm significantly improves the performance of existing detectors, particularly in unseen complex categories, across diverse and challenging scenarios. To facilitate future research, we will publicly release our code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。