利用扩散模型生成轨迹恢复语义分割信息,无需训练即达顶尖性能。
MAVISEG: Manifold Propagation and Visual Prototypes for Zero-Shot Open-Vocabulary Segmentation in Diffusion Transformers

- 基于生成过程的时空结构与特征几何,重构像素级语义信号。
- 在六个基准上均超越现有无训练方法,所有任务达最高mIoU。
- 对弱初始预测提升最大,适合处理复杂或模糊场景的零样本分割。
文本到图像的扩散变换模型通过学习生成物体和场景来获取知识,是实现免训练零样本开放词汇语义分割的理想候选。当前最先进的归因方法独立评估每个像素,将其特征与固定文本导出的类别表示进行比较,无论是输出空间相似性还是交叉注意力权重。这忽略了模型自身暴露的结构性信号:生成轨迹的时间结构、概念的视觉外观统计以及图像内部像素特征的成对几何关系。我们提出MAVISEG,一种免训练的精炼层,用于恢复这些信号。由于其算子仅需像素-概念得分场和像素特征空间,MAVISEG具有捕获无关性,不依赖特定归因方法。在六个基准上,其整体表现优于所有免训练方法,且在每个基准上均取得最佳的mIoU。有趣的是,提升幅度在初始捕获最弱处最大,且各算子贡献取决于待精炼场中的噪声水平。结果表明,扩散变换模型蕴含的语义信息远超现有归因方法所挖掘,大量信息是在生成掩码前丢失,而非模型本身缺失。
原文摘要 · Abstract (English)
Text-to-image diffusion transformers learn about objects and scenes by learning to generate them, making them strong candidates for training-free zero-shot open-vocabulary semantic segmentation. State-of-the-art attribution methods score each pixel independently, comparing its features against a fixed text-derived class representation, whether as an output-space similarity or as a cross-attention weight. This discards structured signals the model itself exposes: the temporal structure of the generative trajectory, the visual appearance statistics of each concept, and the image's own pairwise feature geometry. We present MAVISEG, a training-free refinement layer that recovers these signals. Because its operators consume only a pixel-by-concept score field and a pixel feature space, MAVISEG is capture-agnostic rather than tied to one attribution method. Across six benchmarks it achieves the strongest overall results among training-free methods, including the best mIoU on every benchmark. Interestingly, gains are largest where the initial capture is weakest, and individual operators contribute depending on the noise in the field they refine. Our results indicate that diffusion transformers carry more concept-level information than current attribution methods recover, and that much of it is lost on the way to the mask rather than absent from the model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。