arXiv:2409.18442cs.LGcs.CV2024-09NeurIPS被引 5

无需梯度的解码器反演方法,大幅降低显存与计算时间。

Gradient-free Decoder Inversion in Latent Diffusion Models

  • 提出无梯度反演算法,避免高显存消耗的梯度计算。
  • 在16帧视频生成任务中实现4倍以上加速,显存占用减少70%。
  • 适用于噪声空间水印等实际应用,精度接近传统方法。

在潜在扩散模型(LDMs)中,去噪过程在低维潜在空间进行,解码器用于将潜在表示转换为像素空间表示。尽管通常假设解码器存在精确的逆编码器,但实际中这种精确配对很少存在,而许多应用需要高精度的解码器反演。以往方法借鉴生成对抗网络的反演思路,采用基于梯度的方法,但这类方法在更大潜在空间中需要更多显存和更长计算时间。例如,当前视频LDM可生成超过16帧,但24GB显存的GPU仅能对4帧执行梯度反演。本文提出一种高效的无梯度解码器反演方法,适用于多种潜在模型。理论分析证明了该方法在前向步法及惯性Krasnoselskii-Mann(KM)迭代下的收敛性,且在满足近年LDMs常见的共强制性假设下成立。结合Adam优化器与学习率调度的无梯度方法显著降低计算时间和内存使用,在噪声空间水印等应用中实现高效计算,同时保持与传统方法相当的误差水平。

原文摘要 · Abstract (English)

In latent diffusion models (LDMs), denoising diffusion process efficiently takes place on latent space whose dimension is lower than that of pixel space. Decoder is typically used to transform the representation in latent space to that in pixel space. While a decoder is assumed to have an encoder as an accurate inverse, exact encoder-decoder pair rarely exists in practice even though applications often require precise inversion of decoder. Prior works for decoder inversion in LDMs employed gradient descent inspired by inversions of generative adversarial networks. However, gradient-based methods require larger GPU memory and longer computation time for larger latent space. For example, recent video LDMs can generate more than 16 frames, but GPUs with 24 GB memory can only perform gradient-based decoder inversion for 4 frames. Here, we propose an efficient gradient-free decoder inversion for LDMs, which can be applied to diverse latent models. Theoretical convergence property of our proposed inversion has been investigated not only for the forward step method, but also for the inertial Krasnoselskii-Mann (KM) iterations under mild assumption on cocoercivity that is satisfied by recent LDMs. Our proposed gradient-free method with Adam optimizer and learning rate scheduling significantly reduced computation time and memory usage over prior gradient-based methods and enabled efficient computation in applications such as noise-space watermarking while achieving comparable error levels.

扩散模型无梯度优化解码器反演视频生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。