用未掩码区域生成初始噪声,提升扩散模型修复一致性。
IS-Diff: Improving Diffusion-Based Inpainting with Better Initial Seed
- 从未掩码区域采样初始噪声,模拟被遮挡区域分布
- 动态调整初始化强度,有效减少修复不一致问题
- 无需训练,通用性强,适合大遮挡修复任务
扩散模型在自由形式图像修复中表现优异。然而,原始扩散过程采用随机初始噪声,可能导致被遮挡区域语义不匹配,造成修复结果不一致、与未遮挡区域不连贯等问题。为此,我们提出完全无需训练的IS-Diff方法,通过从未掩码区域采样初始种子来模拟遮挡区域的数据分布,为扩散过程设定更合理的起点。同时引入动态选择性精炼机制,在中间潜空间检测严重不协调修复结果,并动态调整初始化先验强度。我们在CelebA-HQ、ImageNet和Places2数据集上验证了该方法在标准和大遮挡修复任务中的有效性,相比现有最佳方法在各项指标上均有提升。
原文摘要 · Abstract (English)
Diffusion models have shown promising results in free-form inpainting. Recent studies based on refined diffusion samplers or novel architectural designs led to realistic results and high data consistency. However, random initialization seed (noise) adopted in vanilla diffusion process may introduce mismatched semantic information in masked regions, leading to biased inpainting results, e.g., low consistency and low coherence with the other unmasked area. To address this issue, we propose the Initial Seed refined Diffusion Model (IS-Diff), a completely training-free approach incorporating distributional harmonious seeds to produce harmonious results. Specifically, IS-Diff employs initial seeds sampled from unmasked areas to imitate the masked data distribution, thereby setting a promising direction for the diffusion procedure. Moreover, a dynamic selective refinement mechanism is proposed to detect severe unharmonious inpaintings in intermediate latent and adjust the strength of our initialization prior dynamically. We validate our method on both standard and large-mask inpainting tasks using the CelebA-HQ, ImageNet, and Places2 datasets, demonstrating its effectiveness across all metrics compared to state-of-the-art inpainting methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。