用CLIP的图文知识提升目标检测,支持零样本识别新物体。
VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio

- 通过构建高质量正样本增强图文对齐信号
- 在LVIS上零样本检测达36.3~38.1 AP,优于现有方法
- 适合需要识别未见类别的开放词汇检测场景
视觉语言模型如CLIP可为开放词汇目标检测提供丰富的语义先验。然而,如何有效融合文本与视觉知识仍具挑战。本文提出VL-DINO,通过更高效利用CLIP的图文知识改进DINO框架。首先设计查询引导正样本构建(QPSC)模块,生成高质量正样本,增强跨异构数据源训练的适应性并提供更强图文对齐信号;其次引入视觉语义编码器(VSE),将CLIP视觉知识蒸馏至骨干网络特征中,生成融合特征供后续编码器优化;最后通过对象区域语义对齐(ORSA)模块提取以对象为中心的区域特征,并与对应文本嵌入对齐,进一步融入文本线索。在零样本设置下,VL-DINO-T和VL-DINO-L在LVIS基准上分别取得36.3和38.1的AP,持续领先现有先进方法。大量实验验证了该设计的有效性与竞争力。
原文摘要 · Abstract (English)
Vision-language models like CLIP can provide rich semantic priors for open-vocabulary object detection. However, jointly integrating both textual and visual knowledge into detection architectures remains challenging. In this paper, we propose VL-DINO, an open-vocabulary detector that enhances DINO through more effective exploitation of CLIP's vision-language knowledge. Specifically, a Query-guided Positive Sample Construction (QPSC) module is first developed to construct additional high-quality positive samples, enabling the vanilla DINO framework to better accommodate mixed training across heterogeneous data sources while providing more vision-language alignment signals, thereby incorporating richer textual knowledge during training. A Visual Semantic Encoder (VSE) module is then introduced to distill CLIP visual knowledge into backbone-extracted features, producing fused features for subsequent encoder refinement. Based on the fused features, an Object-Region Semantic Alignment (ORSA) module extracts object-centric region features and aligns them with the corresponding textual embeddings, further incorporating textual cues. In the zero-shot setting, VL-DINO-T and VL-DINO-L achieve 36.3 and 38.1 AP on the LVIS benchmark, respectively, consistently outperforming prior advanced approaches. Extensive experiments demonstrate the effectiveness and competitive performance of the proposed design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。