直接在像素空间建模,实现图像生成与理解的统一优化。
Pixel-Space Diffusion Transformers

- 用Transformer直接处理原始像素,跳过编码器瓶颈。
- 支持端到端训练,生成质量更高且结构更精细。
- 适合构建图文统一的多模态基础模型,适合研究生成与理解融合者。
潜在扩散模型(LDMs)通过在VAE压缩的潜在空间中去噪,实现了高效的高分辨率图像生成。然而,固定的视觉分词器会丢失细粒度纹理和结构细节,且重建与生成目标之间的训练不匹配限制了性能。这促使人们重新关注像素空间扩散模型:直接建模原始像素,消除VAE瓶颈,并支持端到端优化。该方法更契合高保真生成需求,但面临高维建模挑战,包括噪声调度、损失加权、令牌效率及可扩展架构设计。此外,像素空间建模为统一多模态系统提供了可能:原始像素、文本和任务条件可共享同一令牌空间,由单一Transformer联合处理,缩小视觉理解与生成之间的差距。本文从模型架构、连续生成机制和统一多模态建模角度综述像素空间扩散Transformer(pDiTs),总结代表性方法,识别关键技术挑战,并讨论向高保真、端到端视觉基础模型发展的未来方向。
原文摘要 · Abstract (English)
Latent diffusion models (LDMs) enable efficient high-resolution image synthesis by denoising in a VAE-compressed latent space. However, fixed visual tokenizers can discard fine textures and structural details, while separate representation and diffusion training creates a mismatch between reconstruction and generation objectives. These limitations have renewed interest in pixel-space diffusion, which models raw pixels directly, removes the VAE bottleneck, and supports end-to-end optimization. This formulation better matches the demands of high-fidelity generation but introduces challenges in high-dimensional modeling, including noise scheduling, loss weighting, token efficiency, and scalable architecture design. Pixel-space modeling also offers a promising basis for unified multimodal systems: raw pixels, text, and task conditions can be represented in a shared token space and jointly processed by a single Transformer, narrowing the gap between visual understanding and generation. This paper reviews Pixel-Space Diffusion Transformers (pDiTs) from the perspectives of model architecture, continuous generative mechanisms, and unified multimodal modeling. We summarize representative methods, identify key technical challenges, and discuss future directions toward high-fidelity, end-to-end vision foundation models that integrate generation and understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。