提出新注意力机制,让扩散Transformer生成更清晰图像。
Dynamic Differential Linear Attention: Enhancing Linear Diffusion Transformer for High-Quality Image Generation
- 设计动态投影与差分操作,缓解线性注意力过平滑问题。
- 在多个数据集上优于当前最优模型,生成图像更清晰细节更丰富。
- 适合关注高效高质图像生成的研究者与开发者。
扩散Transformer(DiTs)在高质量图像生成中表现强劲,但自注意力的二次计算开销成为可扩展性的主要瓶颈。为解决此问题,已有研究采用线性注意力机制降低计算成本,但导致线性扩散Transformer(LiTs)生成质量下降,常出现过度平滑的注意力权重,限制表达能力。本文提出动态差分线性注意力(DyDiLA),通过三项创新设计提升LiTs性能:(i) 动态投影模块,通过动态分配知识解耦标记表示;(ii) 动态度量核,动态分配核函数以捕捉细粒度语义差异;(iii) 标记差分算子,通过计算标记与其冗余信息的差异,实现更鲁棒的查询-键检索。基于DyDiLA,我们构建了改进型LiT——DyDi-LiT。大量实验表明,DyDi-LiT在多个指标上持续超越现有最先进模型,展现出显著的实用潜力。
原文摘要 · Abstract (English)
Diffusion transformers (DiTs) have emerged as a powerful architecture for high-fidelity image generation, yet the quadratic cost of self-attention poses a major scalability bottleneck. To address this, linear attention mechanisms have been adopted to reduce computational cost; unfortunately, the resulting linear diffusion transformers (LiTs) models often come at the expense of generative performance, frequently producing over-smoothed attention weights that limit expressiveness. In this work, we introduce Dynamic Differential Linear Attention (DyDiLA), a novel linear attention formulation that enhances the effectiveness of LiTs by mitigating the oversmoothing issue and improving generation quality. Specifically, the novelty of DyDiLA lies in three key designs: (i) dynamic projection module, which facilitates the decoupling of token representations by learning with dynamically assigned knowledge; (ii) dynamic measure kernel, which provides a better similarity measurement to capture fine-grained semantic distinctions between tokens by dynamically assigning kernel functions for token processing; and (iii) token differential operator, which enables more robust query-to-key retrieval by calculating the differences between the tokens and their corresponding information redundancy produced by dynamic measure kernel. To capitalize on DyDiLA, we introduce a refined LiT, termed DyDi-LiT, that systematically incorporates our advancements. Extensive experiments show that DyDi-LiT consistently outperforms current state-of-the-art (SOTA) models across multiple metrics, underscoring its strong practical potential.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。