发现扩散模型中未被评估的激活信号,提出通用筛选方法提升特征性能
Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features
- 系统评估扩散模型中被忽视的查询、键等激活信号
- 发现三类通用激活特性,可提前过滤低效特征
- 在多个任务上优于现有方法,适合视觉特征提取场景
扩散模型最初用于图像生成,近期研究发现其骨干网络中的内部信号(激活)可作为密集特征用于语义分割等判别任务。面对众多激活信号,如何选出少量高效子集仍是核心挑战。早期研究虽开展大规模定量比较,但许多潜在激活(如注意力计算中的查询、键)未被评估。此外,新型架构(如嵌入式ViT模块)引入大量新激活,使选择问题更加复杂且长期被忽视。本文扩展评估范围,涵盖更广泛的激活类型。由于激活数量激增,全量对比不再可行,转而研究激活的内在属性,通过简单定性分析提前剔除明显劣质信号。经过深入分析,发现三类普遍存在于扩散模型中的激活特性,为跨模型筛选提供依据。在此基础上,提出针对主流扩散模型的有效特征选择方案。多任务实验验证了该方法在性能上优于当前最优基线。代码已开源。
原文摘要 · Abstract (English)
Diffusion models are initially designed for image generation. Recent research shows that the internal signals within their backbones, named activations, can also serve as dense features for various discriminative tasks such as semantic segmentation. Given numerous activations, selecting a small yet effective subset poses a fundamental problem. To this end, the early study of this field performs a large-scale quantitative comparison of the discriminative ability of the activations. However, we find that many potential activations have not been evaluated, such as the queries and keys used to compute attention scores. Moreover, recent advancements in diffusion architectures bring many new activations, such as those within embedded ViT modules. Both combined, activation selection remains unresolved but overlooked. To tackle this issue, this paper takes a further step with a much broader range of activations evaluated. Considering the significant increase in activations, a full-scale quantitative comparison is no longer operational. Instead, we seek to understand the properties of these activations, such that the activations that are clearly inferior can be filtered out in advance via simple qualitative evaluation. After careful analysis, we discover three properties universal among diffusion models, enabling this study to go beyond specific models. On top of this, we present effective feature selection solutions for several popular diffusion models. Finally, the experiments across multiple discriminative tasks validate the superiority of our method over the SOTA competitors. Our code is available at https://github.com/Darkbblue/generic-diffusion-feature.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。