通过优化潜在空间偏差,提升扩散模型在真实图像重建中的保真度。
Latent Bias Alignment for High-Fidelity Diffusion Inversion in Real-World Image Reconstruction and Manipulation
- 引入潜在偏差向量,对齐生成与反演轨迹
- 联合优化扩散反演与VQ自编码器重建,提升图像质量
- 适用于高保真图像重建与编辑,适合图像生成研究者
近期研究表明,文本到图像的扩散模型可基于文本提示生成高质量图像。但能否从种子噪声中生成或逼近真实世界图像?这被称为扩散反演问题,是连接扩散模型与真实场景的基础。然而现有方法常存在重建质量低或鲁棒性差的问题。主要挑战有二:(1)扩散过程中反演与生成轨迹的错位;(2)反演过程与VQ自编码器(VQAE)重建之间的不匹配。为此,我们在每一步反演中引入一个潜在偏差向量,以减小反演与生成轨迹的偏差,称为潜在偏差优化(LBO)。此外,通过学习调整图像潜在表示,实现扩散反演与VQAE重建的近似联合优化,作为两者之间的桥梁,称为图像潜在增强(ILB)。大量实验表明,该方法显著提升了扩散模型的图像重建质量,并增强了下游任务性能,包括图像编辑和罕见概念生成。
原文摘要 · Abstract (English)
Recent research has shown that text-to-image diffusion models are capable of generating high-quality images guided by text prompts. But can they be used to generate or approximate real-world images from the seed noise? This is known as the diffusion inversion problem, which serves as a fundamental building block for bridging diffusion models and real-world scenarios. However, existing diffusion inversion methods often suffer from low reconstruction quality or weak robustness. Two major challenges need to be carefully addressed: (1) the misalignment between the inversion and generation trajectories during the diffusion process, and (2) the mismatch between the diffusion inversion process and the VQ autoencoder (VQAE) reconstruction. To address these challenges, we introduce a latent bias vector at each inversion step, which is learned to reduce the misalignment between inversion and generation trajectories. We refer to this strategy as Latent Bias Optimization (LBO). Furthermore, we perform an approximate joint optimization of the diffusion inversion and VQAE reconstruction processes by learning to adjust the image latent representation, which serves as the connecting interface between them. We refer to this technique as Image Latent Boosting (ILB). Extensive experimental results demonstrate that the proposed method significantly improves the image reconstruction quality of the diffusion model, as well as the performance of downstream tasks, including image editing and rare concept generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。