用VAE隐空间统一预训练，提升图像生成与理解性能
USP: Unified Self-Supervised Pretraining for Image Generation and Understanding
- 在VAE隐空间中通过掩码建模初始化扩散模型
- 生成质量与收敛速度显著提升，理解任务表现相当
- 适合想同时优化生成与理解的视觉模型研究者
近期研究揭示了扩散模型与表征学习之间的相互作用。扩散模型的中间表征可用于下游视觉任务，而自监督视觉模型可提升扩散模型的收敛速度与生成质量。然而，由于输入不匹配及潜在空间差异，将视觉模型预训练权重迁移到扩散模型面临挑战。为此，我们提出统一自监督预训练（USP），通过变分自编码器（VAE）隐空间中的掩码潜在建模来初始化扩散模型。USP在理解任务上达到相当性能的同时，显著提升了扩散模型的收敛速度与生成质量。代码将公开于https://github.com/AMAP-ML/USP。
原文摘要 · Abstract (English)
Recent studies have highlighted the interplay between diffusion models and representation learning. Intermediate representations from diffusion models can be leveraged for downstream visual tasks, while self-supervised vision models can enhance the convergence and generation quality of diffusion models. However, transferring pretrained weights from vision models to diffusion models is challenging due to input mismatches and the use of latent spaces. To address these challenges, we propose Unified Self-supervised Pretraining (USP), a framework that initializes diffusion models via masked latent modeling in a Variational Autoencoder (VAE) latent space. USP achieves comparable performance in understanding tasks while significantly improving the convergence speed and generation quality of diffusion models. Our code will be publicly available at https://github.com/AMAP-ML/USP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。