arXiv:2410.20158cs.CVcs.LG2024-10

用伪视频提升图像生成质量,让模型学会从残损图像中恢复细节。

Your Image is Secretly the Last Frame of a Pseudo Video

  • 将原始图像通过数据增强构造伪视频,提供中间状态自监督信号。
  • 在CIFAR10和CelebA上,生成图像质量显著优于基线模型。
  • 适合对扩散模型机制感兴趣的研究者或图像生成优化开发者。

扩散模型可视为一种特殊的分层变分自编码器(HVAE),在生成逼真图像方面表现卓越。相比之下,标准HVAE生成图像质量较差。本文假设扩散模型的成功部分源于其对中间潜在状态的额外自监督信息——由原始图像与受损图像构成的伪视频。基于此,我们探索如何利用伪视频改进其他生成模型。具体而言,先将图像生成模型扩展为视频生成模型,并在通过数据增强构建的伪视频上进行训练。此外,我们分析了一阶马尔可夫型数据增强方法的局限性,提出使用更具表达力的增强策略以生成更有效的伪视频信息。在CIFAR10和CelebA数据集上的实验证明,引入伪视频的自监督信息可有效提升图像生成质量。

原文摘要 · Abstract (English)

Diffusion models, which can be viewed as a special case of hierarchical variational autoencoders (HVAEs), have shown profound success in generating photo-realistic images. In contrast, standard HVAEs often produce images of inferior quality compared to diffusion models. In this paper, we hypothesize that the success of diffusion models can be partly attributed to the additional self-supervision information for their intermediate latent states provided by corrupted images, which along with the original image form a pseudo video. Based on this hypothesis, we explore the possibility of improving other types of generative models with such pseudo videos. Specifically, we first extend a given image generative model to their video generative model counterpart, and then train the video generative model on pseudo videos constructed by applying data augmentation to the original images. Furthermore, we analyze the potential issues of first-order Markov data augmentation methods, which are typically used in diffusion models, and propose to use more expressive data augmentation to construct more useful information in pseudo videos. Our empirical results on the CIFAR10 and CelebA datasets demonstrate that improved image generation quality can be achieved with additional self-supervised information from pseudo videos.

图像生成伪视频自监督扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。