将地面图像的开放词汇检测能力迁移到航拍图像,实现无需训练即可识别新物体。
Cross-View Open-Vocabulary Object Detection in Aerial Imagery
- 通过对比图像对齐和多实例文本关联,实现地面与航拍图像表征对齐。
- 在零样本设置下,多个数据集上比微调模型提升超3.46点mAP。
- 适合需要快速扩展类别、不依赖标注的新场景航拍检测应用。
传统目标检测模型通常在固定类别上训练,难以灵活扩展新类。开放词汇检测通过无需显式训练即可识别未见类别,缓解此问题。利用大量地面视角图像-文本对预训练的模型,为航拍图像中的开放词汇检测提供强大基础。然而,领域偏移、视角变化和极端尺度差异导致跨域知识迁移失效,需专门适配策略。本文提出一种新颖框架,通过结构化领域对齐,将地面视角的开放词汇表示适配至航拍图像。方法引入对比图像-图像对齐以增强航拍与地面视角嵌入相似性,并采用多实例词汇关联将航拍图像与文本嵌入对齐。在xView、DOTAv2、VisDrone、DIOR和HRRSD数据集上进行充分实验验证。所提开放词汇模型在零样本设置下,相比微调后的封闭词汇专用模型,分别取得+6.32 mAP(DOTAv2)、+4.16 mAP(VisDrone图像)和+3.46 mAP(HRRSD)的提升,为航拍应用中更灵活、可扩展的目标检测系统铺平道路。
原文摘要 · Abstract (English)
Traditional object detection models are typically trained on a fixed set of classes, limiting their flexibility and making it costly to incorporate new categories. Open-vocabulary object detection addresses this limitation by enabling models to identify unseen classes without explicit training. Leveraging pretrained models contrastively trained on abundantly available ground-view image-text classification pairs provides a strong foundation for open-vocabulary object detection in aerial imagery. Domain shifts, viewpoint variations, and extreme scale differences make direct knowledge transfer across domains ineffective, requiring specialized adaptation strategies. In this paper, we propose a novel framework for adapting open-vocabulary representations from ground-view images to solve object detection in aerial imagery through structured domain alignment. The method introduces contrastive image-to-image alignment to enhance the similarity between aerial and ground-view embeddings and employs multi-instance vocabulary associations to align aerial images with text embeddings. Extensive experiments on the xView, DOTAv2, VisDrone, DIOR, and HRRSD datasets are used to validate our approach. Our open-vocabulary model achieves improvements of +6.32 mAP on DOTAv2, +4.16 mAP on VisDrone (Images), and +3.46 mAP on HRRSD in the zero-shot setting when compared to finetuned closed-vocabulary dataset-specific model performance, thus paving the way for more flexible and scalable object detection systems in aerial applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。