arXiv:2505.18584cs.CV2025-05NeurIPS被引 20

通过调控海量激活值,提升扩散Transformer在视觉对应任务中的表现。

Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations

  • 基于AdaLN设计通道调制机制,定位并归一化关键激活值。
  • 在Spair-71k和AP-10K-C.S.上分别提升9.4%和4.4%,达新最优。
  • 无需训练,适用于各类视觉对应任务,尤其适合追求高精度的场景。

预训练的稳定扩散模型(SD)在视觉对应任务中表现出色。本文研究了扩散Transformer(DiTs)在稠密对应任务中的能力。与SD不同,DiTs存在一种关键现象:极少数特征激活值远高于其他值,称为‘海量激活’,导致表征信息不足,性能显著下降。这些激活值集中在固定少数维度,缺乏局部语义信息。我们分析发现,其集中性与DiTs中的自适应层归一化(AdaLN)机制密切相关。基于此,提出无需训练的DiT特征提取框架DiTF,利用AdaLN进行通道级调制,实现对海量激活的自适应定位与归一化,并引入通道丢弃策略缓解其负面影响。实验表明,DiTF在多个视觉对应任务中优于DINO及基于SD的模型,在Spair-71k上提升9.4%,在AP-10K-C.S.上提升4.4%,达到当前最佳水平。

原文摘要 · Abstract (English)

Pre-trained stable diffusion models (SD) have shown great advances in visual correspondence. In this paper, we investigate the capabilities of Diffusion Transformers (DiTs) for accurate dense correspondence. Distinct from SD, DiTs exhibit a critical phenomenon in which very few feature activations exhibit significantly larger values than others, known as \textit{massive activations}, leading to uninformative representations and significant performance degradation for DiTs. The massive activations consistently concentrate at very few fixed dimensions across all image patch tokens, holding little local information. We analyze these dimension-concentrated massive activations and uncover that their concentration is inherently linked to the Adaptive Layer Normalization (AdaLN) in DiTs. Building on these findings, we propose the \textbf{Di}ffusion \textbf{T}ransformer \textbf{F}eature (DiTF), a training-free AdaLN-based framework that extracts semantically discriminative features from DiTs. Specifically, DiTF leverages AdaLN to adaptively localize and normalize massive activations through channel-wise modulation. Furthermore, a channel discard strategy is introduced to mitigate the adverse effects of massive activations. Experimental results demonstrate that our DiTF outperforms both DINO and SD-based models and establishes a new state-of-the-art performance for DiTs in different visual correspondence tasks (\eg, with +9.4\% on Spair-71k and +4.4\% on AP-10K-C.S.).

扩散模型视觉对应特征提取AdaLN

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。