不训练即可实现开放词汇语义分割，提升全局上下文感知能力
OV-Stitcher: A Global Context-Aware Framework for Training-Free Open-Vocabulary Semantic Segmentation
- 通过拼接子图像特征，在编码器末尾重建全局注意力
- 在8个基准上实现mIoU从48.7提升至50.7
- 适合希望零训练部署语义分割的开发者
训练自由的开放词汇语义分割（TF-OVSS）因其无需额外训练即可利用大规模视觉语言模型的预训练知识进行密集预测而受到关注。然而，由于预训练编码器输入分辨率有限，现有方法多采用滑动窗口策略独立处理裁剪子图像，导致无法对全图进行全局注意力，造成特征碎片化和上下文推理受限。本文提出OV-Stitcher，一种训练自由框架，通过在最终编码器块内直接拼接碎片化子图像特征，重建注意力表征，实现最终编码器内的全局注意力，从而生成连贯的上下文聚合与空间一致、语义对齐的分割图。在八个基准上的广泛评估表明，该方法显著提升了开放词汇分割性能，相比先前的无训练基线，平均交并比（mIoU）从48.7提升至50.7。
原文摘要 · Abstract (English)
Training-free open-vocabulary semantic segmentation(TF-OVSS) has recently attracted attention for its ability to perform dense prediction by leveraging the pretrained knowledge of large vision and vision-language models, without requiring additional training. However, due to the limited input resolution of these pretrained encoders, existing TF-OVSS methods commonly adopt a sliding-window strategy that processes cropped sub-images independently. While effective for managing high-resolution inputs, this approach prevents global attention over the full image, leading to fragmented feature representations and limited contextual reasoning. We propose OV-Stitcher, a training-free framework that addresses this limitation by stitching fragmented sub-image features directly within the final encoder block. By reconstructing attention representations from fragmented sub-image features, OV-Stitcher enables global attention within the final encoder block, producing coherent context aggregation and spatially consistent, semantically aligned segmentation maps. Extensive evaluations across eight benchmarks demonstrate that OV-Stitcher establishes a scalable and effective solution for open-vocabulary segmentation, achieving a notable improvement in mean Intersection over Union(mIoU) from 48.7 to 50.7 compared with prior training-free baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。