用视觉样本生成文本化特征，提升模型对罕见目标的检测能力。
Visual Textualization for Image Prompted Object Detection
- 将视觉样本映射到文本空间，生成引导检测的文本化视觉标记。
- 在少样本场景下显著提升PASCAL VOC和MSCOCO的检测性能。
- 保持原模型架构，适合需要强泛化能力的开放集检测任务。
我们提出VisTex-OVLM，一种新型图像提示目标检测方法，引入视觉文本化机制——将少量视觉样例投影至文本特征空间，增强对象级视觉语言模型（OVLM）对难以文字描述且预训练数据中几乎缺失的罕见类别检测能力，同时保留其预训练的物体-文本对齐特性。具体而言，VisTex-OVLM采用多尺度文本化模块与多阶段融合策略，整合视觉样例中的视觉信息，生成可指导OVLM的文本化视觉标记，与文本提示协同工作。不同于以往方法，本方法保持了OVLM原有架构，既维持其泛化能力，又在少样本设置下表现更优。在开放集数据集上表现出色，且在少样本基准PASCAL VOC与MSCOCO上达到当前最佳性能。代码将发布于https://github.com/WitGotFlg/VisTex-OVLM。
原文摘要 · Abstract (English)
We propose VisTex-OVLM, a novel image prompted object detection method that introduces visual textualization -- a process that projects a few visual exemplars into the text feature space to enhance Object-level Vision-Language Models' (OVLMs) capability in detecting rare categories that are difficult to describe textually and nearly absent from their pre-training data, while preserving their pre-trained object-text alignment. Specifically, VisTex-OVLM leverages multi-scale textualizing blocks and a multi-stage fusion strategy to integrate visual information from visual exemplars, generating textualized visual tokens that effectively guide OVLMs alongside text prompts. Unlike previous methods, our method maintains the original architecture of OVLM, maintaining its generalization capabilities while enhancing performance in few-shot settings. VisTex-OVLM demonstrates superior performance across open-set datasets which have minimal overlap with OVLM's pre-training data and achieves state-of-the-art results on few-shot benchmarks PASCAL VOC and MSCOCO. The code will be released at https://github.com/WitGotFlg/VisTex-OVLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。