提出新扩散模型,让图文修复既保原图又对齐文本语义。
One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting
- 分频段解耦修复:按高低频分别处理,避免信息干扰。
- 双阶段修复:先稳中频对齐语义,再低频重建遮蔽区。
- 适合需要精细修复且保持原图的图像编辑场景。
文本引导图像修复旨在根据文本提示重建被遮挡区域,长期挑战在于保持未遮挡区域不变,同时确保遮挡与未遮挡区域语义一致。以往方法难以兼顾两者,根源在于混合(如中低)频带纠缠,其在去噪过程中对文本提示的响应差异显著。本文提出一种零文本-零频率感知扩散模型NTN-Diff,将遮挡与未遮挡区域间的语义一致性分解为各频段独立一致性任务,同时保留未遮挡区域。基于扩散过程,将去噪分为早期(高层噪声)和晚期(低层噪声)阶段,在去噪中解耦中低频带。观察发现,稳定的中频带在文本引导下逐步对齐语义,同时作为引导信号用于零文本去噪低频带,最后在晚期进行文本引导去噪,实现中低频带间语义一致,同时保留未遮挡区域。大量实验验证了NTN-Diff在主流扩散模型上的优越性。
原文摘要 · Abstract (English)
Text-guided image inpainting aims at reconstructing the masked regions as per text prompts, where the longstanding challenges lie in the preservation for unmasked regions, while achieving the semantics consistency between unmasked and inpainted masked regions. Previous arts failed to address both of them, always with either of them to be remedied. Such facts, as we observed, stem from the entanglement of the hybrid (e.g., mid-and-low) frequency bands that encode varied image properties, which exhibit different robustness to text prompts during the denoising process. In this paper, we propose a null-text-null frequency-aware diffusion models, dubbed \textbf{NTN-Diff}, for text-guided image inpainting, by decomposing the semantics consistency across masked and unmasked regions into the consistencies as per each frequency band, while preserving the unmasked regions, to circumvent two challenges in a row. Based on the diffusion process, we further divide the denoising process into early (high-level noise) and late (low-level noise) stages, where the mid-and-low frequency bands are disentangled during the denoising process. As observed, the stable mid-frequency band is progressively denoised to be semantically aligned during text-guided denoising process, which, meanwhile, serves as the guidance to the null-text denoising process to denoise low-frequency band for the masked regions, followed by a subsequent text-guided denoising process at late stage, to achieve the semantics consistency for mid-and-low frequency bands across masked and unmasked regions, while preserve the unmasked regions. Extensive experiments validate the superiority of NTN-Diff over the state-of-the-art diffusion models to text-guided diffusion models. Our code can be accessed from https://github.com/htyjers/NTN-Diff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。