用极少数据和资源实现更优的开放式物体检测,自动生成物体名称。
Open-Det: An Efficient Learning Framework for Open-Ended Detection
- 重构检测与命名模块,加速训练过程。
- 仅用1.5%数据、20.8%轮次,性能反超现有模型1.0%。
- 适合追求高效、低资源开放检测的应用场景。
开放式物体检测(OED)是一项新挑战任务,需在无额外词汇表条件下自由生成物体类别名。现有模型如GenerateU依赖大规模数据训练,收敛慢且性能有限。为此,我们提出高效Open-Det框架,包含四个协同组件:通过重构检测器与命名生成器加速边界框与名称生成;设计视觉-语言对齐器,融合V-to-L与L-to-V机制,并结合提示蒸馏器,将视觉语言模型知识迁移到文本提示中,提升大模型命名准确性;引入掩码对齐损失消除矛盾监督,联合损失增强分类能力。相比GenerateU,Open-Det仅使用0.077M(1.5%)训练数据、31轮(20.8%)训练周期,且4块V100显卡,性能反而提升1.0%(APr)。代码已开源。
原文摘要 · Abstract (English)
Open-Ended object Detection (OED) is a novel and challenging task that detects objects and generates their category names in a free-form manner, without requiring additional vocabularies during inference. However, the existing OED models, such as GenerateU, require large-scale datasets for training, suffer from slow convergence, and exhibit limited performance. To address these issues, we present a novel and efficient Open-Det framework, consisting of four collaborative parts. Specifically, Open-Det accelerates model training in both the bounding box and object name generation process by reconstructing the Object Detector and the Object Name Generator. To bridge the semantic gap between Vision and Language modalities, we propose a Vision-Language Aligner with V-to-L and L-to-V alignment mechanisms, incorporating with the Prompts Distiller to transfer knowledge from the VLM into VL-prompts, enabling accurate object name generation for the LLM. In addition, we design a Masked Alignment Loss to eliminate contradictory supervision and introduce a Joint Loss to enhance classification, resulting in more efficient training. Compared to GenerateU, Open-Det, using only 1.5% of the training data (0.077M vs. 5.077M), 20.8% of the training epochs (31 vs. 149), and fewer GPU resources (4 V100 vs. 16 A100), achieves even higher performance (+1.0% in APr). The source codes are available at: https://github.com/Med-Process/Open-Det.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。