arXiv:2607.29122cs.CV2026-07被引 1

用自生成样本指导冻结的像素扩散模型,显著提升图像质量。

A Frozen Pixel-Space Diffusion Model Can Guide Itself with Its Own Samples

论文配图:A Frozen Pixel-Space Diffusion Model Can Guide Itself with Its Own Samples
图 1 · 摘自论文原文
  • 在预训练模型中间层加轻量预测头,利用前后层差异作为自引导方向。
  • 仅用生成样本训练头,计算量不足全模型的1%,FID降低超50%。
  • 适合想低成本优化像素级扩散模型的研究者与开发者。

像素空间扩散模型旨在直接在原始像素上学习端到端生成器,但需在同一高维空间中同时捕捉全局结构与局部纹理,极具挑战性。尽管近期工作通过改进预测目标、训练目标和架构提升了性能,通常仍需从头训练新模型。本文提出一种更高效互补策略:冻结的预训练像素扩散模型可自我引导。关键观察是,预训练模型的中间层可解码为粗粒度结构预测,而最终层逐步细化高频细节。因此,在中间层附加轻量预测头,保持主干冻结,采样时以中间与最终预测的差异作为自引导方向。训练该头时发现,真实图像非必需;模型生成样本即可,且在增强像素扩散常低估的高频成分上表现更优。在ImageNet上的多个像素扩散模型验证中,我们的合成自引导(SSG)方法一致提升生成质量,适配器训练所需算力低于全模型训练的1%:无分类器自由引导(CFG)的JiT系列模型FID降低超50%;有CFG的强基线亦进一步提升,如JiT-H/16从1.86降至1.67,PixelREPA-H/16从1.81降至1.59。代码已开源。

原文摘要 · Abstract (English)

Pixel-space diffusion models aim to learn an end-to-end generator directly over raw pixels. This is challenging because a single model must capture both global structure and local texture in the same high-dimensional space. While recent work improves pixel diffusion through alternative prediction targets, training objectives, and architectures, these advances typically require training a new model from scratch. We show there is a cheaper, complementary strategy: \textbf{a frozen, pretrained pixel diffusion model can guide itself}. Our key observation is that intermediate layers of a pretrained pixel diffusion transformer can be decoded into coarse predictions that capture the main low-frequency structure, while the final layers progressively refine local, high-frequency details. We therefore attach a lightweight prediction head to an intermediate layer, keep the backbone frozen, and use the discrepancy between the intermediate and final predictions as a self-guidance direction during sampling. To train this head, we further find that real images are not necessary. Instead, model-generated samples suffice and even outperform real images for training the head, especially in enhancing the high-frequency components that pixel diffusion tends to underfit. Across multiple pixel diffusion models on ImageNet, our \textbf{Synthetic Self-Guidance (SSG)} consistently improves generation while adapter training requires less than 1$\%$ of full-model training compute: it reduces FID by over 50$\%$ across the evaluated JiT variants without classifier-free guidance (CFG) and further improves strong baselines with CFG, e.g., JiT-H/16 from 1.86 to 1.67 and PixelREPA-H/16 from 1.81 to 1.59. Our code is available at https://github.com/zfu006/SSG.

扩散模型自引导像素生成低资源

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。