用一步反演提升少步扩散修复,让图像更真实、文字更连贯。
InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting
- 输入掩码图生成语义对齐的初始噪声,替代随机噪声
- 仅1次反演+少步采样,质量接近专用修复模型
- 无需真实图像监督,推理开销极小,适合快速部署
基于扩散的图像修复虽能生成逼真结果,但需大量采样步骤,限制实际应用。少步文本到图像模型虽快,但直接用于修复会导致背景与修复区域不协调、出现伪影。我们发现根源在于低函数评估次数下随机高斯噪声初始化导致语义错位和保真度下降。为此,提出InverFill,一种专为修复设计的一步反演方法,将输入掩码图像的语义信息注入初始噪声中,实现高质量少步修复。InverFill不训练修复模型,而是将少步文本到图像模型与语义对齐噪声结合,在混合采样流程中显著优于基线方法,甚至在低函数评估次数(NFE)下达到专用修复模型水平。此外,InverFill无需真实图像监督,仅增加微小推理开销。大量实验表明,InverFill持续提升基线少步模型性能,显著改善图像质量与文本一致性,无需昂贵重训练或复杂迭代优化。
原文摘要 · Abstract (English)
Recent diffusion-based models achieve photorealism in image inpainting but require many sampling steps, limiting practical use. Few-step text-to-image models offer faster generation, but naively applying them to inpainting yields poor harmonization and artifacts between the background and inpainted region. We trace this cause to random Gaussian noise initialization, which under low function evaluations causes semantic misalignment and reduced fidelity. To overcome this, we propose InverFill, a one-step inversion method tailored for inpainting that injects semantic information from the input masked image into the initial noise, enabling high-fidelity few-step inpainting. Instead of training inpainting models, InverFill leverages few-step text-to-image models in a blended sampling pipeline with semantically aligned noise as input, significantly improving vanilla blended sampling and even matching specialized inpainting models at low NFEs. Moreover, InverFill does not require real-image supervision and only adds minimal inference overhead. Extensive experiments show that InverFill consistently boosts baseline few-step models, improving image quality and text coherence without costly retraining or heavy iterative optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。