用单阶段框架提升开放词汇语义分割的效率与精度
Effective SAM Combination for Open-Vocabulary Semantic Segmentation
- 将伪提示嵌入SAM解码器，实现端到端高效分割
- 在ADE20K等数据集上超越已有方法，兼顾速度与准确率
- 适合需要快速部署高精度分割的工业应用
开放词汇语义分割旨在对图像中无限类别的像素进行标注。传统方法通过串联强大的掩码生成器（如SAM）与预训练视觉语言模型（如CLIP）实现，但存在计算开销大、内存效率低的问题。本文提出ESC-Net，一种新型单阶段开放词汇分割模型，利用SAM解码器块在高效推理框架中完成类别无关分割。通过将图像-文本相关性生成的伪提示嵌入SAM的可提示分割框架，实现更精准的掩码预测。ESC-Net在ADE20K、PASCAL-VOC和PASCAL-Context等标准基准上表现优异，相比先前方法在效率和准确率上均有提升。全面消融实验进一步验证其在复杂条件下的鲁棒性。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation aims to assign pixel-level labels to images across an unlimited range of classes. Traditional methods address this by sequentially connecting a powerful mask proposal generator, such as the Segment Anything Model (SAM), with a pre-trained vision-language model like CLIP. But these two-stage approaches often suffer from high computational costs, memory inefficiencies. In this paper, we propose ESC-Net, a novel one-stage open-vocabulary segmentation model that leverages the SAM decoder blocks for class-agnostic segmentation within an efficient inference framework. By embedding pseudo prompts generated from image-text correlations into SAM's promptable segmentation framework, ESC-Net achieves refined spatial aggregation for accurate mask predictions. ESC-Net achieves superior performance on standard benchmarks, including ADE20K, PASCAL-VOC, and PASCAL-Context, outperforming prior methods in both efficiency and accuracy. Comprehensive ablation studies further demonstrate its robustness across challenging conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。