用大模型生成图像描述，让目标检测器识别更开放词汇
LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models
- 用大语言模型生成图像级详细描述，监督检测器训练
- 在GroundingCap-1M数据集上，性能超越基线明显提升
- 适合做开放词汇检测与多模态模型协同优化的研究者
近期的开放词汇检测器在大量区域级标注数据下表现良好。本文提出，通过大语言模型为每张图像生成图像级详细描述，并与检测器联合训练，可进一步提升性能。为此，我们构建了包含图像级详细描述与对应定位标签的数据集GroundingCap-1M。基于该数据集，对开放词汇检测器进行微调，采用标准定位损失和描述生成损失双重目标。利用大语言模型生成每个感兴趣区域的短描述及整图长描述，在其监督下，所提出的LLMDet显著优于基线，展现出更强的开放词汇能力。此外，改进后的LLMDet还能反向增强多模态模型，形成良性循环。代码、模型与数据集已开源。
原文摘要 · Abstract (English)
Recent open-vocabulary detectors achieve promising performance with abundant region-level annotated data. In this work, we show that an open-vocabulary detector co-training with a large language model by generating image-level detailed captions for each image can further improve performance. To achieve the goal, we first collect a dataset, GroundingCap-1M, wherein each image is accompanied by associated grounding labels and an image-level detailed caption. With this dataset, we finetune an open-vocabulary detector with training objectives including a standard grounding loss and a caption generation loss. We take advantage of a large language model to generate both region-level short captions for each region of interest and image-level long captions for the whole image. Under the supervision of the large language model, the resulting detector, LLMDet, outperforms the baseline by a clear margin, enjoying superior open-vocabulary ability. Further, we show that the improved LLMDet can in turn build a stronger large multi-modal model, achieving mutual benefits. The code, model, and dataset is available at https://github.com/iSEE-Laboratory/LLMDet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。