用概念提示提升DETR的通用目标检测能力,零样本表现更强。
CP-DETR: Concept Prompt Guide DETR Toward Stronger Universal Object Detection
- 设计混合编码器融合文本与视觉提示,增强信息交互
- 在LVIS上达47.6零样本AP,ODinW35上达32.2零样本AP
- 支持交互式检测与稳定对齐,适合开放集检测场景
当前通用目标检测研究致力于将语言引入先进闭集检测器,并通过构建大规模(文本-区域)数据集进行训练以泛化开放集概念。然而,此类方法面临两大挑战:如何高效利用提示中的先验信息实现对象泛化,以及如何降低下游任务中的对齐偏差,导致预训练外场景性能不佳。为此,我们提出强通用检测基础模型CP-DETR,仅需一套预训练权重即可在几乎所有场景中保持竞争力。具体地,设计了高效的提示视觉混合编码器,通过逐级与多尺度融合模块增强提示与视觉信息的交互。结合提示多标签损失和辅助检测头,充分挖掘提示信息。此外,提出两种实用的概念提示生成方法——视觉提示与优化提示,分别通过具体视觉样例提取抽象概念,稳定降低下游任务对齐偏差。实验表明,采用Swin-T主干的模型在LVIS上实现47.6零样本AP,Swin-L主干在ODinW35上达32.2零样本AP;视觉提示方法在COCO val上通过交互检测达68.4 AP,优化提示在ODinW13上实现73.1全量样本AP。
原文摘要 · Abstract (English)
Recent research on universal object detection aims to introduce language in a SoTA closed-set detector and then generalize the open-set concepts by constructing large-scale (text-region) datasets for training. However, these methods face two main challenges: (i) how to efficiently use the prior information in the prompts to genericise objects and (ii) how to reduce alignment bias in the downstream tasks, both leading to sub-optimal performance in some scenarios beyond pre-training. To address these challenges, we propose a strong universal detection foundation model called CP-DETR, which is competitive in almost all scenarios, with only one pre-training weight. Specifically, we design an efficient prompt visual hybrid encoder that enhances the information interaction between prompt and visual through scale-by-scale and multi-scale fusion modules. Then, the hybrid encoder is facilitated to fully utilize the prompted information by prompt multi-label loss and auxiliary detection head. In addition to text prompts, we have designed two practical concept prompt generation methods, visual prompt and optimized prompt, to extract abstract concepts through concrete visual examples and stably reduce alignment bias in downstream tasks. With these effective designs, CP-DETR demonstrates superior universal detection performance in a broad spectrum of scenarios. For example, our Swin-T backbone model achieves 47.6 zero-shot AP on LVIS, and the Swin-L backbone model achieves 32.2 zero-shot AP on ODinW35. Furthermore, our visual prompt generation method achieves 68.4 AP on COCO val by interactive detection, and the optimized prompt achieves 73.1 fully-shot AP on ODinW13.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。