无需训练即可实现开放词汇图像分割与识别,融合高效网络与视觉语言模型。
A Training-Free Framework for Open-Vocabulary Image Segmentation and Recognition with EfficientNet and CLIP
- 用EfficientNetB0+奇异值分解做无监督像素分块
- 通过CLIP对分割区域进行开放词汇识别,达最优性能
- 完全免训练,适合快速部署于新类别场景
本文提出一种无需训练的开放词汇图像分割与识别框架(OVSR),结合EfficientNetB0用于无监督分割,CLIP用于开放词汇识别。第一阶段:从EfficientNetB0提取像素特征,经奇异值分解得到潜在表示,再通过层次聚类分割出语义合理区域,聚类数量由奇异值分布自适应确定。第二阶段:使用CLIP的视觉变压器主干将分割区域编码为图像嵌入,文本嵌入则通过CLIP文本编码器预先计算,包含类别特定提示和通用“其他”提示以支持开放集识别。图像与文本嵌入拼接后经奇异值分解投影至共享潜空间,提升跨模态对齐,最终通过计算相似度的softmax完成识别。在COCO、ADE20K和PASCAL VOC等标准数据集上评估,匈牙利匹配平均交并比(Hungarian mIoU)、精度、召回率和F1分数均达到当前最优水平,验证了该框架的有效性、灵活性与泛化能力。
原文摘要 · Abstract (English)
This paper presents a novel training-free framework for open-vocabulary image segmentation and object recognition (OVSR), which leverages EfficientNetB0, a convolutional neural network, for unsupervised segmentation and CLIP, a vision-language model, for open-vocabulary object recognition. The proposed framework adopts a two stage pipeline: unsupervised image segmentation followed by segment-level recognition via vision-language alignment. In the first stage, pixel-wise features extracted from EfficientNetB0 are decomposed using singular value decomposition to obtain latent representations, which are then clustered using hierarchical clustering to segment semantically meaningful regions. The number of clusters is adaptively determined by the distribution of singular values. In the second stage, the segmented regions are localized and encoded into image embeddings using the Vision Transformer backbone of CLIP. Text embeddings are precomputed using CLIP's text encoder from category-specific prompts, including a generic something else prompt to support open set recognition. The image and text embeddings are concatenated and projected into a shared latent feature space via SVD to enhance cross-modal alignment. Recognition is performed by computing the softmax over the similarities between the projected image and text embeddings. The proposed method is evaluated on standard benchmarks, including COCO, ADE20K, and PASCAL VOC, achieving state-of-the-art performance in terms of Hungarian mIoU, precision, recall, and F1-score. These results demonstrate the effectiveness, flexibility, and generalizability of the proposed framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。