arXiv:2411.18207cs.CVcs.AI2024-11中稿 · BMVC 2025被引 9

让视觉语言模型在开放世界中识别新物体,解决误检和漏检问题。

From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects

  • 引入伪未知嵌入与多尺度对比锚点学习,定位未知物体。
  • 在COCO和nuScenes上达到领先性能,同时保持开放词汇检测能力。
  • 适合自动驾驶等需可靠感知新物体的高风险场景。

传统目标检测方法基于封闭集假设,仅能识别训练集中预定义的物体。近期开放词汇检测(OVD)虽支持无限词汇量的物体识别,但严重依赖精确提示,限制了其在驾驶场景等关键应用中的使用。现有OVD模型常将近域分布外(NOOD)物体误判为已知类,忽略远域分布外(FOOD)物体。为此,本文提出一种框架,使OVD模型能在开放世界中检测并逐步学习未见物体。针对食品级物体检测,提出开放世界嵌入学习(OWEL),通过已知类信息推断未知类在连续语义空间中的位置;提出多尺度对比锚点学习(MSCAL),在不同尺度下增强物体嵌入的类内一致性,以识别被误分类的未知物体。该方法在标准开放世界检测与自动驾驶基准测试中均达到当前最优表现,同时保留开放词汇检测能力。

原文摘要 · Abstract (English)

Traditional object detection methods operate under the closed-set assumption, where models can only detect a fixed number of objects predefined in the training set. Recent works on open vocabulary object detection (OVD) enable the detection of objects defined by an in-principle unbounded vocabulary, which reduces the cost of training models for specific tasks. However, OVD heavily relies on accurate prompts provided by an ``oracle'', which limits their use in critical applications such as driving scene perception. OVD models tend to misclassify near-out-of-distribution (NOOD) objects that have similar features to known classes, and ignore far-out-of-distribution (FOOD) objects. To address these limitations, we propose a framework that enables OVD models to operate in open world settings, by identifying and incrementally learning previously unseen objects. To detect FOOD objects, we propose Open World Embedding Learning (OWEL) and introduce the concept of Pseudo Unknown Embedding which infers the location of unknown classes in a continuous semantic space based on the information of known classes. We also propose Multi-Scale Contrastive Anchor Learning (MSCAL), which enables the identification of misclassified unknown objects by promoting the intra-class consistency of object embeddings at different scales. The proposed method achieves state-of-the-art performance on standard open world object detection and autonomous driving benchmarks while maintaining its open vocabulary object detection capability.

开放世界检测视觉语言模型目标检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。