用简单结构实现高效像素扩散,性能超越复杂模型。
PixelU: A U-Shaped Transformer for Efficient End-to-End Pixel Diffusion

- 设计单阶段U形架构,用零成本跳接保留高频细节。
- 在ImageNet上256×256和512×512分别达FID 1.63和1.92。
- 计算量仅为基线1/3,适合资源受限场景
端到端像素空间扩散模型绕过潜在扩散模型的有损压缩,但在高维空间中难以同时建模低频语义与高频信号。现有方法依赖复杂的像素解码器缓解此问题。本文揭示这些解码器主要补偿速度预测(v-prediction)的优化困难,在纯净数据设定(x-prediction)下实为冗余。基于此洞察,我们倡导简洁优于复杂,提出PixelU——一种面向像素空间的极简单阶段U形扩散变压器。PixelU摒弃辅助解码器,改用零成本跳跃连接,形成“信息高速公路”,直接传递未经污染的高频空间细节。为使主干专注建模低频语义,引入恒定通道的空间下采样机制,作为天然低通滤波器,将深层特征压缩为紧凑的低频语义流形。大量实验表明,这种频率解耦可使模型以约1/3的计算成本超越强基线JiT-G。在ImageNet 256×256和512×512上,PixelU分别取得FID 1.63和1.92,超越近期像素空间方法,建立了一种简单而强大的新范式。
原文摘要 · Abstract (English)
End-to-end pixel-space diffusion models bypass the lossy compression of Latent Diffusion Models (LDMs) but struggle to jointly model low-frequency semantics and high-frequency signals in high-dimensional space. Existing works heavily rely on complex pixel decoders to alleviate this issue. In this paper, we challenge this trend by revealing that these decoders primarily compensate for the optimization difficulties inherent to velocity prediction ($v$-prediction). Under the clean data paradigm ($x$-prediction), they are redundant. Motivated by this insight, we advocate for simplicity over complexity and introduce PixelU, a minimalist, single-stage U-shaped Diffusion Transformer tailored for pixel space. PixelU abandons auxiliary decoders in favor of zero-cost skip connections, which provide an "information highway" that directly routes uncorrupted high-frequency spatial details from shallow to deep layers. To further enable the backbone to focus exclusively on modeling low-frequency semantics, we introduce a constant-channel spatial down-sampling mechanism as a natural low-pass filter, which compresses deep features into a compact, low-frequency semantic manifold. Extensive experiments demonstrate that this decoupling of frequencies could outperform the strong baseline (JiT-G) with only about 1/3 of its computation cost. On ImageNet 256$\times$256 and 512$\times$512, PixelU achieves FID of 1.63 and 1.92 respectively, surpassing recent pixel-space methods and establishing a simple yet powerful new paradigm for end-to-end diffusion models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。