用场景图提升未知物体检测,让模型理解物体间关系。
Unveiling the Unknown: Open Vocabulary Object Detection with Scene Graphs

- 引入场景图捕捉物体间语义与空间关系。
- 在COCO和LVIS上新类别检测准确率显著提升。
- 适合需要理解复杂场景关系的视觉检测任务。
开放词汇物体检测旨在识别训练数据中未出现的新物体类别。许多基于知识蒸馏的方法通过将预训练视觉语言模型的知识迁移到检测任务中,取得了良好效果。然而,这些方法常忽略图像中物体间的结构化关系,如交互与空间布局,从而限制了对新类别的检测能力。为此,我们提出一种场景引导的关系建模检测框架。该框架利用场景图捕捉候选区域与其上下文物体之间的结构化语义与空间关系,显式建模邻近区域间的交互,并引入关系注意力模块隐式强化从场景图中提取的关键关系线索。此外,我们设计了一个基于场景的文本对齐分支,从图像描述中蒸馏类别知识以指导关系对齐。该方法实现了视觉关系与语义信息的无缝融合,显著提升了检测性能。大量实验表明,我们的模型在COCO和LVIS数据集上对新类别物体的平均精度(AP)均优于现有开放词汇检测方法。
原文摘要 · Abstract (English)
Open-vocabulary object detection seeks to identify novel object categories that were not part of the training data. Many knowledge distillation-based approaches have shown promising performance by transferring knowledge from pre-trained vision-language models to object detection. However, these methods often overlook structured, image-specific relationships between objects, such as interactions and spatial arrangements. This oversight can significantly restrict the effectiveness of detecting novel categories. To address this issue, we propose a Scene-guided Relational Modeling detection framework. This framework utilizes scene graphs to capture structured semantic and spatial relationships between candidate regions and their contextual objects. It explicitly models interactions among neighboring regions and incorporates a Relation Attention Module to implicitly amplify the key relational cues extracted from the scene graph. Furthermore, we present a scene-based textual alignment branch that distills category knowledge from captions to guide relational alignment. This approach facilitates a seamless integration of visual relations with semantic information for enhanced detection performance. Comprehensive experiments show that our model achieves superior performance compared to other OVOD methods, improving the AP for novel categories on COCO and LVIS datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。