arXiv:2412.04678cs.CV2024-12中稿 · The IEEE/CVF Winte…被引 2

利用扩散模型注意力机制实现零样本图像分割,无需训练即可达到顶尖效果。

Unsupervised Segmentation by Diffusing, Walking and Cutting

  • 基于自注意力构建随机游走模型,通过归一化割法递归分割图像。
  • 在COCO-Stuff-27和Cityscapes上超越所有现有无监督方法,达当前最佳。
  • 自动确定分割阈值,避免人工调参,适合无标注数据场景使用。

我们提出一种基于预训练文本到图像扩散模型特征的无监督图像分割方法。受经典谱聚类启发,从图像块间的自注意力层构建邻接矩阵,并采用归一化割(NCuts)进行递归划分。关键洞察是:自注意力概率分布可视为图像上随机游走的转移矩阵。我们直接在自注意力激活值上应用随机游走归一化割,最小化簇间转移概率,最大化簇内一致性。递归执行得到反映预训练注意力语义层次的分层分割,无需额外训练。进一步探索了不同特征构造邻接矩阵的方式,以及利用随机游走捕捉长程关系的方法。最后提出自动确定NCuts代价准则的方案,避免手动调参。定量分析了不同特征、固定与动态阈值、多节点路径对邻接矩阵的影响。结果表明,该方法在零样本无监督分割任务中全面领先,于COCO-Stuff-27和Cityscapes上取得当前最优性能。

原文摘要 · Abstract (English)

We propose an unsupervised image segmentation method using features from pre-trained text-to-image diffusion models. Inspired by classic spectral clustering approaches, we construct adjacency matrices from self-attention layers between image patches and recursively partition using Normalised Cuts. A key insight is that self-attention probability distributions, which capture semantic relations between patches, can be interpreted as a transition matrix for random walks across the image. We leverage this by first using Random Walk Normalized Cuts directly on these self-attention activations to partition the image, minimizing transition probabilities between clusters while maximizing coherence within clusters. Applied recursively, this yields a hierarchical segmentation that reflects the rich semantics in the pre-trained attention layers, without any additional training. Next, we explore other ways to build the NCuts adjacency matrix from features, and how we can use the random walk interpretation of self-attention to capture long-range relationships. Finally, we propose an approach to automatically determine the NCut cost criterion, avoiding the need to tune this manually. We quantitatively analyse the effect incorporating different features, a constant versus dynamic NCut threshold, and incorporating multi-node paths when constructing the NCuts adjacency matrix. We show that our approach surpasses all existing methods for zero-shot unsupervised segmentation, achieving state-of-the-art results on COCO-Stuff-27 and Cityscapes.

无监督分割扩散模型自注意力随机游走

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。