用生成样本提取视觉先验,提升扩散模型的细节生成能力
Diffusion Models Need Visual Priors for Image Generation
- 多阶段框架从已有生成样例中提取视觉先验
- 仅用100万步训练,FID-50K达1.83,性能超越现有方法
- 适合关注图像细节生成与训练效率的研究者
传统类别引导的扩散模型虽能生成正确语义内容,但在纹理细节上表现不佳,根源在于仅使用类别先验提供的粗粒度信息。为此,我们提出扩散对扩散(DoD)框架,通过多阶段生成,从前期生成样本中提取视觉先验,并在扩散采样早期提供丰富引导。具体地,引入潜空间嵌入模块,采用压缩-重建策略去除条件样本中的冗余细节,仅保留语义信息用于指导。我们在ImageNet-256×256数据集上评估,相比SiT和DiT,训练成本降低7倍,且在FID-50K指标上表现更优。最大模型DoD-XL仅用100万次训练步数即达FID-50K=1.83,无需推理时额外技巧,超越当前最优方法。
原文摘要 · Abstract (English)
Conventional class-guided diffusion models generally succeed in generating images with correct semantic content, but often struggle with texture details. This limitation stems from the usage of class priors, which only provide coarse and limited conditional information. To address this issue, we propose Diffusion on Diffusion (DoD), an innovative multi-stage generation framework that first extracts visual priors from previously generated samples, then provides rich guidance for the diffusion model leveraging visual priors from the early stages of diffusion sampling. Specifically, we introduce a latent embedding module that employs a compression-reconstruction approach to discard redundant detail information from the conditional samples in each stage, retaining only the semantic information for guidance. We evaluate DoD on the popular ImageNet-$256 \times 256$ dataset, reducing 7$\times$ training cost compared to SiT and DiT with even better performance in terms of the FID-50K score. Our largest model DoD-XL achieves an FID-50K score of 1.83 with only 1 million training steps, which surpasses other state-of-the-art methods without bells and whistles during inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。