在频域生成图像，用DCTdiff实现高效高质量生成。
DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space
- 直接在DCT频域构建扩散模型，避免像素空间建模。
- 512×512图像生成仅需1/4训练成本，超越潜空间扩散模型。
- 揭示频域生成的理论机制，适合对生成模型效率有要求的研究者。
本文探索在频率空间进行图像建模的可能性，提出DCTdiff——一种端到端的频域扩散生成范式，能在离散余弦变换（DCT）空间高效建模图像。通过系统研究DCTdiff的设计空间，揭示关键设计因素。在不同框架（UViT、DiT）、生成任务及多种扩散采样器上的实验表明，DCTdiff在生成质量与训练效率上均优于基于像素的扩散模型。尤为显著的是，DCTdiff无需潜空间扩散架构即可无缝扩展至512×512分辨率，且训练成本仅为潜空间扩散模型（使用SD-VAE）的1/4。此外，本文揭示了若干有趣的DCT图像建模特性，例如从理论上证明‘图像扩散可视为谱自回归’，弥合了扩散模型与自回归模型之间的鸿沟。DCTdiff的有效性及其引入的新特性，为频域图像建模指明了新方向。代码已开源：https://github.com/forever208/DCTdiff。
原文摘要 · Abstract (English)
This paper explores image modeling from the frequency space and introduces DCTdiff, an end-to-end diffusion generative paradigm that efficiently models images in the discrete cosine transform (DCT) space. We investigate the design space of DCTdiff and reveal the key design factors. Experiments on different frameworks (UViT, DiT), generation tasks, and various diffusion samplers demonstrate that DCTdiff outperforms pixel-based diffusion models regarding generative quality and training efficiency. Remarkably, DCTdiff can seamlessly scale up to 512$\times$512 resolution without using the latent diffusion paradigm and beats latent diffusion (using SD-VAE) with only 1/4 training cost. Finally, we illustrate several intriguing properties of DCT image modeling. For example, we provide a theoretical proof of why 'image diffusion can be seen as spectral autoregression', bridging the gap between diffusion and autoregressive models. The effectiveness of DCTdiff and the introduced properties suggest a promising direction for image modeling in the frequency space. The code is https://github.com/forever208/DCTdiff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。