通过视觉与文本原型融合,提升跨域少样本目标检测精度
Learning Multi-Modal Prototypes for Cross-Domain Few-Shot Object Detection
- 用目标域图像构建视觉原型,结合文本提示实现多模态引导
- 在六组跨域数据集上1/5/10-shot设置下达最优或接近最优mAP
- 适合需要跨域适应的少样本检测场景,尤其关注定位精度
跨域少样本目标检测(CD-FSOD)旨在仅用少量标注样本的情况下,在未见目标域中检测新类别。尽管基于视觉-语言模型(VLMs)的开放词汇检测器具备良好迁移能力,但其几乎完全依赖文本提示,虽能表达领域无关语义,却缺失目标域特有的视觉信息,导致在少样本监督下定位不准确。本文提出双分支检测器LMP(Learning Multi-modal Prototypes),将文本指导与来自目标域的视觉样例相结合。视觉原型构建模块从支持集区域(support RoIs)聚合类级别原型,并通过扰动框在查询图像中动态生成硬负样本原型,捕捉干扰物与视觉相似背景。在视觉引导分支中,将这些原型注入检测流程,其组件沿用文本分支结构作为训练起点;同时并行的文本引导分支保留开放词汇语义。两分支联合训练,推理时通过融合语义抽象与领域自适应细节进行集成。在六个跨域基准数据集及标准1/5/10-shot设置下,本方法在平均精度(mAP)上达到当前最优或高度竞争力。
原文摘要 · Abstract (English)
Cross-Domain Few-Shot Object Detection (CD-FSOD) aims to detect novel classes in unseen target domains given only a few labeled examples. While open-vocabulary detectors built on vision-language models (VLMs) transfer well, they depend almost entirely on text prompts, which encode domain-invariant semantics but miss domain-specific visual information needed for precise localization under few-shot supervision. We propose a dual-branch detector that Learns Multi-modal Prototypes, dubbed LMP, by coupling textual guidance with visual exemplars drawn from the target domain. A Visual Prototype Construction module aggregates class-level prototypes from support RoIs and dynamically generates hard-negative prototypes in query images via jittered boxes, capturing distractors and visually similar backgrounds. In the visual-guided branch, we inject these prototypes into the detection pipeline with components mirrored from the text branch as the starting point for training, while a parallel text-guided branch preserves open-vocabulary semantics. The branches are trained jointly and ensembled at inference by combining semantic abstraction with domain-adaptive details. On six cross-domain benchmark datasets and standard 1/5/10-shot settings, our method achieves state-of-the-art or highly competitive mAP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。