无需微调即可高效编辑真实图像,保持结构和局部细节。
Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing

- 用自引导机制在生成时保留原图整体结构与未编辑区域
- 引入布局保真能量函数,提升局部与全局结构一致性
- 无需模型微调或反演,适合快速高质量图像编辑
尽管大规模文本到图像生成模型取得进展,但利用这些模型操控真实图像仍具挑战。现有方法或在多种编辑任务中质量不稳定,或需耗时的超参数调优或扩散模型微调以保持输入图像的特定外观。本文提出一种基于修改后扩散采样过程的自引导方法。通过探索自引导技术,保留输入图像的整体结构及未编辑区域的局部外观。特别地,我们显式引入布局保真能量函数,以保存源图像的局部与全局结构。此外,提出噪声重缩放机制,在生成过程中平衡无分类器引导与所提引导器的范数,从而保持噪声分布。该引导方法无需微调扩散模型或精确反演过程。实验表明,通过人类评估与定量分析,该方法能生成更受人类青睐的编辑结果,并在编辑质量与原图保留之间实现更优权衡。代码已公开于 https://github.com/MACderRu/Guide-and-Rescale。
原文摘要 · Abstract (English)
Despite recent advances in large-scale text-to-image generative models, manipulating real images with these models remains a challenging problem. The main limitations of existing editing methods are that they either fail to perform with consistent quality on a wide range of image edits or require time-consuming hyperparameter tuning or fine-tuning of the diffusion model to preserve the image-specific appearance of the input image. We propose a novel approach that is built upon a modified diffusion sampling process via the guidance mechanism. In this work, we explore the self-guidance technique to preserve the overall structure of the input image and its local regions appearance that should not be edited. In particular, we explicitly introduce layout-preserving energy functions that are aimed to save local and global structures of the source image. Additionally, we propose a noise rescaling mechanism that allows to preserve noise distribution by balancing the norms of classifier-free guidance and our proposed guiders during generation. Such a guiding approach does not require fine-tuning the diffusion model and exact inversion process. As a result, the proposed method provides a fast and high-quality editing mechanism. In our experiments, we show through human evaluation and quantitative analysis that the proposed method allows to produce desired editing which is more preferable by humans and also achieves a better trade-off between editing quality and preservation of the original image. Our code is available at https://github.com/MACderRu/Guide-and-Rescale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。