不靠迭代优化,直接用分布差异解出分割图
Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation

- 跳过传统对数似然优化,直接求解分布差异的解析解
- 在8个数据集上达到顶尖性能,无需训练或注意力调优
- 适合追求高效、无训练开集语义分割的研究者
开集语义分割(OVSS)旨在利用开放词汇提示分割图像中任意类别的区域,要求方法具备像素级视觉-语言对齐能力。通常依赖计算视觉与语言特征间的余弦相似度(即对数似然值),并通过最小化对数似然与真实标签之间的分布差异来优化对数似然,再据此生成分割图,但这一过程需耗时的迭代训练或模型特定的注意力调优。本文提出一种更直接的方法:摒弃对数似然优化,直接推导分割图的解析解。核心假设是:分布差异编码了语义信息——同一类别内块间具有一致性,跨类别则不一致。基于此,直接将该分布差异的解析解作为语义图。即将对分布差异的优化转化为解析解的推导,从而消除耗时的迭代训练,摆脱模型特定的注意力调优,在八个基准数据集上实现最先进性能。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation (OVSS) aims to segment arbitrary category regions in images using open-vocabulary prompts, necessitating that existing methods possess pixel-level vision-language alignment capability. Typically, this capability involves computing the cosine similarity, \ie, logits, between visual and linguistic features, and minimizing the distribution discrepancy between the logits and the ground truth (GT) to generate optimal logits that are subsequently used to construct segmentation maps, yet it depends on time-consuming iterative training or model-specific attention modulation. In this work, we propose a more direct approach that eschews the logits-optimization process by directly deriving an analytic solution for the segmentation map. We posit a key hypothesis: the distribution discrepancy encodes semantic information; specifically, this discrepancy exhibits consistency across patches belonging to the same category but inconsistency across different categories. Based on this hypothesis, we directly utilize the analytic solution of this distribution discrepancy as the semantic maps. In other words, we reformulate the optimization of the distribution discrepancy as deriving its analytic solution, thereby eliminating time-consuming iterative training, freeing us from model-specific attention modulation, and achieving state-of-the-art performance on eight benchmark datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。