用3×3卷积重构扩散模型，速度更快且性能更强
DiC: Rethinking Conv3x3 Designs in Diffusion Models
- 用时序卷积替代自注意力，构建纯卷积扩散架构
- 在多个数据集上超越扩散变压器，生成速度提升40%以上
- 适合追求高效生成的落地应用，如实时图像合成
扩散模型在视觉生成任务中表现卓越。近期模型从传统的CNN-注意力混合结构转向全注意力的各向同性架构，虽具强可扩展性和性能，但依赖复杂自注意力机制导致推理速度慢。本文重新思考深度学习中最简单快速的3×3卷积模块，设计一种可扩展的纯卷积扩散模型。首先发现编码器-解码器沙漏结构优于可扩展的各向同性架构，但仍未达预期；进一步引入稀疏跳连以减少冗余并提升可扩展性。基于此架构，提出阶段特异性嵌入、中间块条件注入和条件门控等条件增强方法，构建出名为DiC的扩散卷积模型。在多种规模与设置下实验表明，DiC在性能上显著超越现有扩散变压器，同时保持优异的速度优势。
原文摘要 · Abstract (English)
Diffusion models have shown exceptional performance in visual generation tasks. Recently, these models have shifted from traditional U-Shaped CNN-Attention hybrid structures to fully transformer-based isotropic architectures. While these transformers exhibit strong scalability and performance, their reliance on complicated self-attention operation results in slow inference speeds. Contrary to these works, we rethink one of the simplest yet fastest module in deep learning, 3x3 Convolution, to construct a scaled-up purely convolutional diffusion model. We first discover that an Encoder-Decoder Hourglass design outperforms scalable isotropic architectures for Conv3x3, but still under-performing our expectation. Further improving the architecture, we introduce sparse skip connections to reduce redundancy and improve scalability. Based on the architecture, we introduce conditioning improvements including stage-specific embeddings, mid-block condition injection, and conditional gating. These improvements lead to our proposed Diffusion CNN (DiC), which serves as a swift yet competitive diffusion architecture baseline. Experiments on various scales and settings show that DiC surpasses existing diffusion transformers by considerable margins in terms of performance while keeping a good speed advantage. Project page: https://github.com/YuchuanTian/DiC
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。