arXiv:2409.19589cs.CV2024-09AAAI被引 37

用Transformer设计扩散模型,实现从零训练的图像超分,效果媲美预训练方法。

Effective Diffusion Transformer Architecture for Image Super-Resolution

论文配图:Effective Diffusion Transformer Architecture for Image Super-Resolution
图 1 · 摘自论文原文
  • 采用统一架构的U型Transformer,支持多尺度特征提取。
  • 在从零训练条件下超越现有扩散模型,逼近预训练方法效果。
  • 改进时间步条件模块,更好处理不同频率信息,适合追求性能的视觉重建研究者。

近期研究表明扩散模型在图像超分辨率中潜力巨大。尽管当前主流方法基于带有卷积网络的潜在扩散模型,但对展现出色生成能力的Transformer探索较少。本文提出一种有效的扩散Transformer用于图像超分辨率(DiT-SR),在无需预训练先验的情况下,实现了与依赖先验的方法相当的视觉质量。DiT-SR采用整体的U型架构,并对各阶段的Transformer模块统一采用各向同性设计,前者促进多尺度层次化特征提取,后者将计算资源集中于关键层以进一步提升性能。此外,我们深入分析了常用AdaLN的局限性,提出一种频率自适应的时间步条件模块,增强模型在不同时间步处理不同频率信息的能力。大量实验表明,DiT-SR显著优于现有的从零训练扩散超分方法,甚至在部分指标上超越使用预训练Stable Diffusion的某些方法,证明了扩散Transformer在图像超分辨率中的优越性。

原文摘要 · Abstract (English)

Recent advances indicate that diffusion models hold great promise in image super-resolution. While the latest methods are primarily based on latent diffusion models with convolutional neural networks, there are few attempts to explore transformers, which have demonstrated remarkable performance in image generation. In this work, we design an effective diffusion transformer for image super-resolution (DiT-SR) that achieves the visual quality of prior-based methods, but through a training-from-scratch manner. In practice, DiT-SR leverages an overall U-shaped architecture, and adopts a uniform isotropic design for all the transformer blocks across different stages. The former facilitates multi-scale hierarchical feature extraction, while the latter reallocates the computational resources to critical layers to further enhance performance. Moreover, we thoroughly analyze the limitation of the widely used AdaLN, and present a frequency-adaptive time-step conditioning module, enhancing the model's capacity to process distinct frequency information at different time steps. Extensive experiments demonstrate that DiT-SR outperforms the existing training-from-scratch diffusion-based SR methods significantly, and even beats some of the prior-based methods on pretrained Stable Diffusion, proving the superiority of diffusion transformer in image super-resolution.

图像超分扩散模型Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。