arXiv:2505.11196cs.CV2025-05NeurIPS被引 8

用卷积网络替代注意力机制,实现高效且高质量的图像生成。

DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling

  • 用紧凑通道注意力增强卷积网络特征多样性。
  • 在ImageNet上512×512图像生成FID达2.53,速度提升3.1倍。
  • 适合追求高效生成模型的研究者和工业应用开发者。

扩散变换器(DiT)在视觉生成中表现优异,但计算开销大。分析发现,其全局自注意力常冗余,主要捕捉局部模式,暗示更高效替代方案的可能。本文重新审视卷积作为构建高效且具表达力扩散模型的替代模块。然而,直接以卷积替换注意力通常导致性能下降。我们发现这是由于卷积网络通道冗余更高。为此,提出一种紧凑通道注意力机制,促进更多样化通道激活,提升特征多样性。由此构建出完全由标准卷积模块组成的扩散卷积网络(DiCo),在类条件ImageNet生成任务中,DiCo-XL在256×256分辨率下取得FID 2.05,512×512下为2.53,分别比DiT-XL/2快2.7倍和3.1倍。此外,在MS-COCO数据集上,纯卷积的DiCo也展现出强大的文本到图像生成潜力。代码已开源。

原文摘要 · Abstract (English)

Diffusion Transformer (DiT), a promising diffusion model for visual generation, demonstrates impressive performance but incurs significant computational overhead. Intriguingly, analysis of pre-trained DiT models reveals that global self-attention is often redundant, predominantly capturing local patterns-highlighting the potential for more efficient alternatives. In this paper, we revisit convolution as an alternative building block for constructing efficient and expressive diffusion models. However, naively replacing self-attention with convolution typically results in degraded performance. Our investigations attribute this performance gap to the higher channel redundancy in ConvNets compared to Transformers. To resolve this, we introduce a compact channel attention mechanism that promotes the activation of more diverse channels, thereby enhancing feature diversity. This leads to Diffusion ConvNet (DiCo), a family of diffusion models built entirely from standard ConvNet modules, offering strong generative performance with significant efficiency gains. On class-conditional ImageNet generation benchmarks, DiCo-XL achieves an FID of 2.05 at 256x256 resolution and 2.53 at 512x512, with a 2.7x and 3.1x speedup over DiT-XL/2, respectively. Furthermore, experimental results on MS-COCO demonstrate that the purely convolutional DiCo exhibits strong potential for text-to-image generation. Code: https://github.com/shallowdream204/DiCo.

扩散模型卷积网络高效生成图像生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。