基于DETR的实时开放词汇检测新方法,提升稀有类别识别能力
OV-DEIM: Real-time DETR-Style Open-Vocabulary Object Detection with GridSynthetic Augmentation
- 采用DEIMv2框架与视觉语言建模,实现端到端开放词汇检测
- 在保持实时推理速度下,稀有类别检测性能显著提升
- 提出网格合成增强策略,改善模型对复杂场景的泛化能力
实时开放词汇目标检测对动态环境中的实际部署至关重要,要求模型在严格延迟限制下识别大量且持续演化的类别。现有实时开放词汇检测方法主要基于YOLO风格模型,而基于DETR的方法在推理延迟、模型轻量化和整体性能上仍落后。本文提出OV-DEIM,一种基于最新DEIMv2框架的端到端DETR式开放词汇检测器,集成视觉语言建模以实现高效开放词汇推理。我们引入简单的查询补全策略,在不牺牲推理速度的前提下提升固定类别平均精度(Fixed AP)。此外,提出名为GridSynthetic的简单但有效的数据增强策略,将多个训练样本组合成结构化图像网格。通过单次前向传播暴露模型于更丰富的物体共现模式与空间布局,缓解噪声定位信号对分类损失的负面影响,显著提升稀有类别的语义区分能力。大量实验表明,OV-DEIM在开放词汇检测基准上达到领先性能,兼具优异效率与对挑战性稀有类别的显著改进。代码与预训练模型已开源。
原文摘要 · Abstract (English)
Real-time open-vocabulary object detection (OVOD) is essential for practical deployment in dynamic environments, where models must recognize a large and evolving set of categories under strict latency constraints. Current real-time OVOD methods are predominantly built upon YOLO-style models. In contrast, real-time DETR-based methods still lag behind in terms of inference latency, model lightweightness, and overall performance. In this work, we present OV-DEIM, an end-to-end DETR-style open-vocabulary detector built upon the recent DEIMv2 framework with integrated vision-language modeling for efficient open-vocabulary inference. We further introduce a simple query supplement strategy that improves Fixed AP without compromising inference speed. Beyond architectural improvements, we introduce GridSynthetic, a simple yet effective data augmentation strategy that composes multiple training samples into structured image grids. By exposing the model to richer object co-occurrence patterns and spatial layouts within a single forward pass, GridSynthetic mitigates the negative impact of noisy localization signals on the classification loss and improves semantic discrimination, particularly for rare categories. Extensive experiments demonstrate that OV-DEIM achieves state-of-the-art performance on open-vocabulary detection benchmarks, delivering superior efficiency and notable improvements on challenging rare categories. Code and pretrained models are available at https://github.com/wleilei/OV-DEIM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。