无需训练，通过像素级标签传播提升开放词汇语义分割精度
LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentation
- 利用视觉模型捕捉图像块间关系，改进VLM初始预测
- 在像素级别进行标签传播，显著提升边界区域分割准确率
- 全图推理避免分块处理，适合需要高精度的开放词汇场景
我们提出一种基于视觉-语言模型（VLM）的无训练开放词汇语义分割方法。该方法通过标签传播机制，联合优化图像块之间的预测关系，以增强VLM的初始每块预测。由于VLM主要优化跨模态对齐而非模态内相似性，我们引入一个更擅长捕捉此类关系的视觉模型（VM）。为克服基于块编码器的分辨率限制，我们在像素级别应用标签传播作为精修步骤，显著提升类边界附近的分割精度。所提方法LPOSS+在整幅图像上进行推理，避免窗口化处理，从而捕获全局上下文交互。LPOSS+在多个数据集上实现了当前无训练方法中的最优性能。代码开源：https://github.com/vladan-stojnic/LPOSS
原文摘要 · Abstract (English)
We propose a training-free method for open-vocabulary semantic segmentation using Vision-and-Language Models (VLMs). Our approach enhances the initial per-patch predictions of VLMs through label propagation, which jointly optimizes predictions by incorporating patch-to-patch relationships. Since VLMs are primarily optimized for cross-modal alignment and not for intra-modal similarity, we use a Vision Model (VM) that is observed to better capture these relationships. We address resolution limitations inherent to patch-based encoders by applying label propagation at the pixel level as a refinement step, significantly improving segmentation accuracy near class boundaries. Our method, called LPOSS+, performs inference over the entire image, avoiding window-based processing and thereby capturing contextual interactions across the full image. LPOSS+ achieves state-of-the-art performance among training-free methods, across a diverse set of datasets. Code: https://github.com/vladan-stojnic/LPOSS
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。