arXiv:2410.19635cs.CV2024-10NeurIPS被引 22

用冻结的视觉大模型提升目标检测性能,无需微调即可增效。

Frozen-DETR: Enhancing DETR with Image Understanding from Frozen Foundation Models

  • 用大模型的类标记提供场景上下文,增强检测器解码能力
  • 用大模型的图像块标记丰富编码器语义特征,提升识别精度
  • 不替换主干网络,插件式接入即可实现性能显著提升

近期视觉基础模型能提取通用表征,在多种任务中表现优异,但其在目标检测中的应用仍被忽视,尤其未进行微调时。本文表明,即使未针对目标检测预训练,冻结的基础模型也可作为多功能特征增强器。我们通过两种方式直接迁移基础模型的高层图像理解能力:首先,基础模型的类标记提供了对复杂场景的深入理解,有助于检测器解码器中对象查询的生成;其次,基础模型的图像块标记可为检测器编码器提供丰富的语义细节。以插件形式使用冻结的基础模型,而非传统主干网络,能显著提升检测器性能,同时避免因架构差异带来的问题。采用此新范式,将基于查询的先进检测器 DINO 在使用 R50 主干、训练 12 轮的 COCO 验证集上,从 49.0% AP 提升至 51.9% AP(+2.9% AP),进一步融合一个或两个基础模型后达到 53.8% AP(+4.8% AP)。

原文摘要 · Abstract (English)

Recent vision foundation models can extract universal representations and show impressive abilities in various tasks. However, their application on object detection is largely overlooked, especially without fine-tuning them. In this work, we show that frozen foundation models can be a versatile feature enhancer, even though they are not pre-trained for object detection. Specifically, we explore directly transferring the high-level image understanding of foundation models to detectors in the following two ways. First, the class token in foundation models provides an in-depth understanding of the complex scene, which facilitates decoding object queries in the detector's decoder by providing a compact context. Additionally, the patch tokens in foundation models can enrich the features in the detector's encoder by providing semantic details. Utilizing frozen foundation models as plug-and-play modules rather than the commonly used backbone can significantly enhance the detector's performance while preventing the problems caused by the architecture discrepancy between the detector's backbone and the foundation model. With such a novel paradigm, we boost the SOTA query-based detector DINO from 49.0% AP to 51.9% AP (+2.9% AP) and further to 53.8% AP (+4.8% AP) by integrating one or two foundation models respectively, on the COCO validation set after training for 12 epochs with R50 as the detector's backbone.

目标检测视觉大模型特征增强DETR

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。