通过灵活引导内部表征,显著加速扩散模型训练并提升生成质量。
Learning Diffusion Models with Flexible Representation Guidance
- 设计双重表征引导策略,融合自洽与跨模态数据对。
- 在图像、蛋白质和分子生成任务中实现23.3倍训练加速。
- 适合追求高效生成与高质量输出的研究者使用。
扩散模型可通过引入额外的表征引导来提升输入表示的有效性。已有实证研究表明,将扩散模型的内部表征与预训练模型对齐可改善生成质量。本文提出一个系统框架,将表征引导融入扩散模型。我们提供了去噪模型的多种分解方式及其对应的训练准则,分解方式决定了辅助表征的引入时机与方式。基于理论洞察,提出两种新策略:一是将样本与其自身或不同合成模态产生的目标表征配对,学习多模态联合模型;二是设计最优训练课程,平衡表征学习与数据生成。在图像、蛋白质序列和分子生成任务上的实验表明,该方法性能优越且训练加速明显。尤其在类条件ImageNet $256 imes 256$ 基准上,训练速度比原始SiT-XL快23.3倍,比当前最优方法REPA快4倍。代码已公开于https://github.com/ChenyuWang-Monica/REED。
原文摘要 · Abstract (English)
Diffusion models can be improved with additional guidance towards more effective representations of input. Indeed, prior empirical work has already shown that aligning internal representations of the diffusion model with those of pre-trained models improves generation quality. In this paper, we present a systematic framework for incorporating representation guidance into diffusion models. We provide alternative decompositions of denoising models along with their associated training criteria, where the decompositions determine when and how the auxiliary representations are incorporated. Guided by our theoretical insights, we introduce two new strategies for enhancing representation alignment in diffusion models. First, we pair examples with target representations either derived from themselves or arisen from different synthetic modalities, and subsequently learn a joint model over the multimodal pairs. Second, we design an optimal training curriculum that balances representation learning and data generation. Our experiments across image, protein sequence, and molecule generation tasks demonstrate superior performance as well as accelerated training. In particular, on the class-conditional ImageNet $256\times 256$ benchmark, our guidance results in $23.3$ times faster training than the original SiT-XL as well as four times speedup over the state-of-the-art method REPA. The code is available at https://github.com/ChenyuWang-Monica/REED.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。