arXiv:2502.14377cs.CV2025-02AAAI被引 19

通过评估控制信息重要性,实现扩散模型的高效可控生成。

RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers

  • 按层评估控制信号重要性,动态调整控制模块配置。
  • 仅用15%参数和计算量,性能优于PixArt-delta。
  • 创新设计二维混洗混合器,提升效率且保持生成质量。

扩散Transformer在文本到图像和文本到视频生成中发挥关键作用,主要得益于其固有的可扩展性。然而,现有受控扩散Transformer方法存在显著的参数和计算开销,且因未能考虑不同变压器层中控制信息的相关性差异,导致资源分配效率低下。为此,我们提出一种相关性引导的高效可控生成框架RelaCtrl,实现扩散变压器中控制信号的高效、资源优化集成。首先,通过评估“ControlNet相关性得分”——即跳过每层控制对生成质量和控制效果的影响——来衡量扩散变压器各层对控制信息的相关性。基于相关性强度,进一步定制控制层的位置、参数规模和建模能力,以减少不必要的参数和冗余计算。此外,为进一步提升效率,我们将常用复制块中的自注意力和前馈网络替换为精心设计的二维混洗混合器(TDSM),实现了令牌混合器和通道混合器的高效实现。定性和定量实验结果表明,该方法在仅使用PixArt-delta 15%的参数和计算复杂度的情况下,仍能达到更优性能。

原文摘要 · Abstract (English)

The Diffusion Transformer plays a pivotal role in advancing text-to-image and text-to-video generation, owing primarily to its inherent scalability. However, existing controlled diffusion transformer methods incur significant parameter and computational overheads and suffer from inefficient resource allocation due to their failure to account for the varying relevance of control information across different transformer layers. To address this, we propose the Relevance-Guided Efficient Controllable Generation framework, RelaCtrl, enabling efficient and resource-optimized integration of control signals into the Diffusion Transformer. First, we evaluate the relevance of each layer in the Diffusion Transformer to the control information by assessing the "ControlNet Relevance Score"-i.e., the impact of skipping each control layer on both the quality of generation and the control effectiveness during inference. Based on the strength of the relevance, we then tailor the positioning, parameter scale, and modeling capacity of the control layers to reduce unnecessary parameters and redundant computations. Additionally, to further improve efficiency, we replace the self-attention and FFN in the commonly used copy block with the carefully designed Two-Dimensional Shuffle Mixer (TDSM), enabling efficient implementation of both the token mixer and channel mixer. Both qualitative and quantitative experimental results demonstrate that our approach achieves superior performance with only 15% of the parameters and computational complexity compared to PixArt-delta.

扩散模型可控生成效率优化Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。