让扩散模型直接预测干净图像，用大块Transformer实现高效生成。
Back to Basics: Let Denoising Generative Models Denoise
- 直接预测干净图像而非噪声，回归数据流形本质。
- 16/32大块像素Transformer在256/512分辨率下达竞品水平。
- 无需分词器、预训练或额外损失，纯像素建模更简洁。
当前的去噪扩散模型并未在经典意义上进行去噪，即不直接预测干净图像，而是预测噪声或带噪量。本文认为预测干净数据与预测带噪量本质不同：根据流形假设，自然数据位于低维流形上，而带噪量则不然。因此我们主张直接预测干净数据，使容量有限的网络也能在高维空间中有效运作。实验表明，仅使用无分词器、无预训练、无额外损失的大块像素Transformer（JiT）即可成为强大生成模型。在ImageNet上，使用16和32大块尺寸，在256和512分辨率下取得可比性能，而传统方法在高维带噪空间中可能失效。本研究回归基础，提出基于Transformer的原始自然数据自包含生成范式。
原文摘要 · Abstract (English)
Today's denoising diffusion models do not "denoise" in the classical sense, i.e., they do not directly predict clean images. Rather, the neural networks predict noise or a noised quantity. In this paper, we suggest that predicting clean data and predicting noised quantities are fundamentally different. According to the manifold assumption, natural data should lie on a low-dimensional manifold, whereas noised quantities do not. With this assumption, we advocate for models that directly predict clean data, which allows apparently under-capacity networks to operate effectively in very high-dimensional spaces. We show that simple, large-patch Transformers on pixels can be strong generative models: using no tokenizer, no pre-training, and no extra loss. Our approach is conceptually nothing more than "Just image Transformers", or JiT, as we call it. We report competitive results using JiT with large patch sizes of 16 and 32 on ImageNet at resolutions of 256 and 512, where predicting high-dimensional noised quantities can fail catastrophically. With our networks mapping back to the basics of the manifold, our research goes back to basics and pursues a self-contained paradigm for Transformer-based diffusion on raw natural data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。