用检索思想实现快速开放词汇目标检测,支持多种下游任务。
WeDetect: Fast Open-Vocabulary Object Detection as Retrieval
- 将目标检测视为文本查询与图像区域的嵌入空间匹配,无融合层设计提升速度。
- 在15个基准上达到顶尖性能,实时推理且支持历史数据对象检索。
- 适配大模型处理复杂指代表达,可集成到多模态系统中使用。
开放词汇目标检测旨在通过文本提示检测任意类别。不使用跨模态融合层(非融合)的方法将识别问题转化为检索任务,即在共享嵌入空间中匹配图像区域与文本查询。本文充分探索此检索范式,提出名为WeDetect的模型家族:(1) 首创实时检测器,采用双塔结构,经精心数据与完整训练后,非融合的WeDetect超越其他融合模型,建立强健的开放词汇检测基础;(2) 实现历史数据快速回溯,WeDetect-Uni为基于WeDetect的通用提议生成器,冻结整个检测器仅微调物体性提示,可跨类别检索通用物体提议,其类别特定的提议嵌入支持新应用——对象检索;(3) 与大语言多模态模型(LMM)结合,提出WeDetect-Ref,用于指代表达理解(REC),从WeDetect-Uni提取的提议列表中检索目标对象,摒弃逐词预测,单次前向传播完成分类。WeDetect系列统一了检测、提议生成、对象检索与指代表达理解,覆盖15个基准,兼具顶尖性能与高推理效率。
原文摘要 · Abstract (English)
Open-vocabulary object detection aims to detect arbitrary classes via text prompts. Methods without cross-modal fusion layers (non-fusion) offer faster inference by treating recognition as a retrieval problem, \ie, matching regions to text queries in a shared embedding space. In this work, we fully explore this retrieval philosophy and demonstrate its unique advantages in efficiency and versatility through a model family named WeDetect: (1) State-of-the-art performance. WeDetect is a real-time detector with a dual-tower architecture. We show that, with well-curated data and full training, the non-fusion WeDetect surpasses other fusion models and establishes a strong open-vocabulary foundation. (2) Fast backtrack of historical data. WeDetect-Uni is a universal proposal generator based on WeDetect. We freeze the entire detector and only finetune an objectness prompt to retrieve generic object proposals across categories. Importantly, the proposal embeddings are class-specific and enable a new application, object retrieval, supporting retrieval objects in historical data. (3) Integration with LMMs for referring expression comprehension (REC). We further propose WeDetect-Ref, an LMM-based object classifier to handle complex referring expressions, which retrieves target objects from the proposal list extracted by WeDetect-Uni. It discards next-token prediction and classifies objects in a single forward pass. Together, the WeDetect family unifies detection, proposal generation, object retrieval, and REC under a coherent retrieval framework, achieving state-of-the-art performance across 15 benchmarks with high inference efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。