修复扩散模型反演噪声缺陷,提升图像编辑质量
There and Back Again: On the relation between Noise and Image Inversions in Diffusion Models
- 用前向扩散替代初始反演步骤,解耦潜在编码
- 反演噪声在平滑区域多样性不足,影响编辑效果
- 改进后实现更自然的图像插值与高质量编辑
扩散模型在生成新样本方面表现卓越,但缺乏可编辑的低维隐空间。基于反演的方法通过逆向去噪轨迹将图像映射到近似初始噪声。本文深入分析该过程,发现反演得到的潜在表示在平滑区域(如纯天空)预测噪声多样性不足。通过一系列分析,我们追溯问题根源在于初始反演步骤无法提供准确且多样的噪声。因此,DDIM反演空间的可编辑性远低于原始噪声。现有反演方法未能完全解决此问题,而我们的简单改进——用前向扩散替换前几轮反演步骤——有效解耦了潜在编码,显著提升编辑和插值质量。代码已公开于 https://github.com/luk-st/taba。
原文摘要 · Abstract (English)
Diffusion Models achieve state-of-the-art performance in generating new samples but lack a low-dimensional latent space that encodes the data into editable features. Inversion-based methods address this by reversing the denoising trajectory, transferring images to their approximated starting noise. In this work, we thoroughly analyze this procedure and focus on the relation between the initial noise, the generated samples, and their corresponding latent encodings obtained through the DDIM inversion. First, we show that latents exhibit structural patterns in the form of less diverse noise predicted for smooth image areas (e.g., plain sky). Through a series of analyses, we trace this issue to the first inversion steps, which fail to provide accurate and diverse noise. Consequently, the DDIM inversion space is notably less manipulative than the original noise. We show that prior inversion methods do not fully resolve this issue, but our simple fix, where we replace the first DDIM Inversion steps with a forward diffusion process, successfully decorrelates latent encodings and enables higher quality editions and interpolations. The code is available at https://github.com/luk-st/taba.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。