用预训练模型生成图像，零数据训练出高分辨率像素生成器
L2P: Unlocking Latent Potential for Pixel Generation
- 用大块令牌化替代VAE，冻结原模型中间层，只训练浅层转换潜空间到像素
- 仅用合成图像训练，8卡即可快速收敛，4K超分辨率生成无内存瓶颈
- 迁移能力强，性能达源模型93%，适合想快速部署高质图像生成的开发者
像素扩散模型近年来重新受到关注。然而，从头训练先进的像素空间模型需要极高的计算和数据资源。为此，我们提出潜空间到像素（L2P）迁移范式，一种高效框架，直接利用预训练扩散模型（LDM）的丰富知识构建强大的像素空间模型。具体而言，L2P摒弃VAE，采用大块令牌化，并冻结源LDM的中间层，仅训练浅层以学习潜空间到像素的映射。通过仅使用LDM生成的合成图像作为训练数据，L2P在已平滑的数据流形上拟合，实现无需真实数据收集的快速收敛。该策略使L2P仅用8张GPU即可无缝将大规模潜空间先验迁移到像素空间。此外，消除VAE内存瓶颈，支持原生4K超高清分辨率生成。在主流LDM架构上的大量实验表明，L2P训练开销极小，但在DPG-Bench上表现与源模型相当，在GenEval上达到93%的性能。
原文摘要 · Abstract (English)
Pixel diffusion models have recently regained attention for visual generation. However, training advanced pixel-space models from scratch demands prohibitive computational and data resources. To address this, we propose the Latent-to-Pixel (L2P) transfer paradigm, an efficient framework that directly harnesses the rich knowledge of pre-trained LDMs to build powerful pixel-space models. Specifically, L2P discards the VAE in favor of large-patch tokenization and freezes the source LDM's intermediate layers, exclusively training shallow layers to learn the latent-to-pixel transformation. By utilizing LDM-generated synthetic images as the sole training corpus, L2P fits an already smooth data manifold, enabling rapid convergence with zero real-data collection. This strategy allows L2P to seamlessly migrate massive latent priors to the pixel space using only 8 GPUs. Furthermore, eliminating the VAE memory bottleneck unlocks native 4K ultra-high resolution generation. Extensive experiments across mainstream LDM architectures show that L2P incurs negligible training overhead, yet performs on par with the source LDM on DPG-Bench and reaches 93% performance on GenEval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。