PixelDiT直接在像素空间生成图像，打破传统两阶段流程，效果超越现有模型。
PixelDiT: Pixel Diffusion Transformers for Image Generation
- 全变压器架构，分层级处理全局语义与局部纹理细节。
- ImageNet 256上FID达1.61，512上为1.81，优于现有像素级生成模型。
- 支持文生图，1024²分辨率预训练，性能接近顶尖潜空间模型。
潜空间建模是扩散Transformer（DiTs）的标准方法，但依赖预训练自编码器的两阶段流程会引入有损重建，导致误差累积并阻碍联合优化。为此，我们提出PixelDiT，一种单阶段、端到端的模型，无需自编码器，直接在像素空间学习扩散过程。PixelDiT采用由双层级设计构成的全变压器架构：局部块级DiT捕捉全局语义，像素级DiT精细化纹理，实现高效像素空间扩散模型训练并保留细节。在ImageNet 256上达到1.61 FID，ImageNet 512上达1.81 FID，超越现有像素生成模型。进一步扩展至文生图任务，在1024²分辨率下像素空间预训练，GenEval得分为0.74，DPG-bench为83.5，逼近最佳潜空间扩散模型。代码已开源。
原文摘要 · Abstract (English)
Latent-space modeling has been the standard for Diffusion Transformers (DiTs). However, it relies on a two-stage pipeline where the pretrained autoencoder introduces lossy reconstruction, leading to error accumulation while hindering joint optimization. To address these issues, we propose PixelDiT, a single-stage, end-to-end model that eliminates the need for the autoencoder and learns the diffusion process directly in the pixel space. PixelDiT adopts a fully transformer-based architecture shaped by a dual-level design: a patch-level DiT that captures global semantics and a pixel-level DiT that refines texture details, enabling efficient training of a pixel-space diffusion model while preserving fine details. PixelDiT achieves 1.61 FID on ImageNet 256 and 1.81 FID on ImageNet 512, surpassing existing pixel generative models. We further extend PixelDiT to text-to-image generation and pretrain it at the 10242resolution in pixel space. It achieves 0.74 on GenEval and 83.5 on DPG-bench, approaching the best latent diffusion models. Code: https://github.com/NVlabs/PixelDiT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。