用分步策略实现无需重训的开放词汇多物体识别，提升泛化能力。
A novel Framework for Open-Vocabulary Multi-Object Recognition using CLIP
- 先分割物体再匹配图文嵌入，流程简洁无需复杂训练
- 基于CLIP的图像编码在COCO等数据集上达到最高平均精度
- 提出独立于CLIP的CNN+MLP编码方法，增强灵活性与可扩展性
为解决现有开放词汇物体识别方法系统复杂度高、训练成本大、泛化能力有限的问题，本文提出一种基于简化两阶段策略（物体分割后识别）的开放词汇物体识别（OVOR）框架。该框架无需复杂重训与人工标注。在提取物体区域后，利用CLIP生成物体级图像嵌入和类别级文本嵌入，实现对任意词汇的识别。为降低对CLIP的依赖并提升编码灵活性，进一步引入基于CNN/MLP的方法：从卷积神经网络提取特征图，并通过多层感知机（MLP）对齐视觉特征与文本嵌入，最终将嵌入拼接进行跨模态匹配。物体识别通过图像与文本嵌入间的相似度匹配完成。在COCO、Pascal VOC和ADE20K上的实验表明，基于CLIP的图像编码达到最高平均精度（AP），优于当前最先进方法；同时揭示了不依赖CLIP的编码方式在OVOR中的潜在优势。
原文摘要 · Abstract (English)
To address the limitations of existing open-vocabulary object recognition methods, including high system complexity, substantial training costs, and limited generalization capability, this paper proposes a novel Open-Vocabulary Object Recognition (OVOR) framework based on a streamlined two-stage strategy: object segmentation followed by recognition. The proposed framework eliminates the need for complex retraining procedures and labor-intensive annotation. After extracting object regions, object-level image embeddings and category-level text embeddings are generated using CLIP, enabling recognition over arbitrary vocabularies. To reduce dependence on CLIP and enhance encoding flexibility, we further introduce a CNN/MLP-based approach that extracts convolutional neural network (CNN) feature maps and employs a multilayer perceptron (MLP) to align visual features with text embeddings. The resulting embeddings are then concatenated for cross-modal matching. Finally, object recognition is performed through similarity matching between image and text embeddings. Experiments on COCO, Pascal VOC, and ADE20K demonstrate that CLIP-based image encoding achieves the highest average AP, outperforming current state-of-the-art methods. Meanwhile, the results reveal the potential of CLIP-independent image encoding as a promising alternative for OVOR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。