arXiv:2410.06719cs.CVcs.AI2024-10NeurIPS被引 3

用现成生成技术抑制扩散模型特征的内容偏移,提升判别性能。

Suppress Content Shift: Better Diffusion Features via Off-the-Shelf Generation Techniques

  • 通过分析去噪过程的信息漂移,定位内容偏移根源。
  • 提出GATE准则,评估并实现现成生成技术对偏移的抑制效果。
  • 在多任务、多数据集上显著提升扩散特征质量,适合特征提取场景。

扩散模型不仅是生成工具,其内部激活值还可作为判别任务的特征(扩散特征)。我们发现扩散特征受限于一种隐蔽且普遍存在的现象——内容偏移:特征与输入图像在物体形状等细节上存在差异。实证研究显示,即使内容偏移不可见,其负面影响仍不可忽略。我们指出内容偏移源于扩散模型固有特性,因此广泛存在。为此,提出抑制内容偏移以提升特征质量。由于内容偏移与从噪声中恢复图像时的信息漂移相关,可将现成的生成技术转化为抑制工具。进一步提出实用指南GATE,高效评估技术收益,并提供实现方案。尽管方法简单,但在多个任务和数据集上表现优异,验证了其作为扩散特征通用增强器的潜力。代码已开源。

原文摘要 · Abstract (English)

Diffusion models are powerful generative models, and this capability can also be applied to discrimination. The inner activations of a pre-trained diffusion model can serve as features for discriminative tasks, namely, diffusion feature. We discover that diffusion feature has been hindered by a hidden yet universal phenomenon that we call content shift. To be specific, there are content differences between features and the input image, such as the exact shape of a certain object. We locate the cause of content shift as one inherent characteristic of diffusion models, which suggests the broad existence of this phenomenon in diffusion feature. Further empirical study also indicates that its negative impact is not negligible even when content shift is not visually perceivable. Hence, we propose to suppress content shift to enhance the overall quality of diffusion features. Specifically, content shift is related to the information drift during the process of recovering an image from the noisy input, pointing out the possibility of turning off-the-shelf generation techniques into tools for content shift suppression. We further propose a practical guideline named GATE to efficiently evaluate the potential benefit of a technique and provide an implementation of our methodology. Despite the simplicity, the proposed approach has achieved superior results on various tasks and datasets, validating its potential as a generic booster for diffusion features. Our code is available at https://github.com/Darkbblue/diffusion-content-shift.

扩散模型特征提取生成技术内容偏移

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。