无需训练的开放词汇分割新方法,通过随机游走优化邻域关系。
NERVE: Neighbourhood & Entropy-guided Random-walk for training free open-Vocabulary sEgmentation
- 基于注意力图熵值筛选关键信息,动态融合多尺度局部特征。
- 不依赖固定核或后处理,实现任意形状物体的精准分割。
- 在7个基准上达到当前最优零样本性能,适合部署于无标注场景。
尽管开放词汇语义分割(OVSS)取得进展,现有免训练方法仍存在计算成本高、注意力图融合效率低、局部平滑约束过于刚性等问题。本文提出一种新的免训练基线方法NERVE(Neighbourhood & Entropy-guided Random-walk for open-Vocabulary sEgmentation),首次结合稳定扩散模型自注意力层的邻域结构,利用熵引导的选择机制,动态筛选最相关的注意力图。引入随机游走策略替代固定大小高斯核,实现空间扩散,促进语义相关区域间的传播,有效分割任意形状目标。相比以往平均加权或多头注意力等方法,本方法避免使用条件随机场(CRF)或像素自适应掩码细化(PAMR)等后处理技术。在7个主流语义分割基准上验证,实现当前最优的零样本分割性能,为开放词汇分割提供了高效实用的新方案。
原文摘要 · Abstract (English)
Despite recent advances in Open-Vocabulary Semantic Segmentation (OVSS), existing training-free methods face several limitations: use of computationally expensive affinity refinement strategies, ineffective fusion of transformer attention maps due to equal weighting or reliance on fixed-size Gaussian kernels to reinforce local spatial smoothness, enforcing isotropic neighborhoods. We propose a strong baseline for training-free OVSS termed as NERVE (Neighbourhood \& Entropy-guided Random-walk for open-Vocabulary sEgmentation), which uniquely integrates global and fine-grained local information, exploiting the neighbourhood structure from the self-attention layer of a stable diffusion model. We also introduce a stochastic random walk for refining the affinity rather than relying on fixed-size Gaussian kernels for local context. This spatial diffusion process encourages propagation across connected and semantically related areas, enabling it to effectively delineate objects with arbitrary shapes. Whereas most existing approaches treat self-attention maps from different transformer heads or layers equally, our method uses entropy-based uncertainty to select the most relevant maps. Notably, our method does not require any conventional post-processing techniques like Conditional Random Fields (CRF) or Pixel-Adaptive Mask Refinement (PAMR). Experiments are performed on 7 popular semantic segmentation benchmarks, yielding an overall state-of-the-art zero-shot segmentation performance, providing an effective approach to open-vocabulary semantic segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。