用可复用的引导模块提升扩散模型图像修复速度与质量。
Amortized Guidance for Image Inpainting with Pretrained Diffusion Models
- 训练小型通用引导模块，无需对每张图单独优化。
- 在多个数据集上均优于现有固定主干和渐进式方法。
- 仅增加不到1%参数量，适合实际部署场景。
我们研究基于生成扩散模型的图像修复。现有方法要么训练专用模型，要么在推理时为每张遮挡图像单独调整预训练模型。本文提出一种折中方案——基于扩散模型的可复用修复（AID），保持预训练扩散主干不变，离线训练一个小型可复用引导模块，并在多种遮挡图像上直接复用，无需逐实例优化。将问题建模为带监督终端目标的确定性引导任务，通过引入辅助高斯形式并证明其随机解可恢复最优确定性引导场，构建了端到端可训练的连续时间演员-评论家算法。实验表明，在AFHQv2、FFHQ（像素级EDM管道）和ImageNet（潜在空间EDM2管道）上，AID在多种掩码类型下持续提升质量-速度权衡，性能优于强基线，且可训练参数增加不足1%。
原文摘要 · Abstract (English)
We study image inpainting with generative diffusion models. Existing methods typically either train dedicated task-specific models, or adapt a pretrained diffusion model separately for each masked image at deployment. We introduce a middle-ground model, termed Amortized Inpainting with Diffusion (AID), which keeps a pretrained diffusion backbone fixed, trains a small reusable guidance module offline, and then reuses it across masked images without per-instance optimization. We formulate it as a deterministic guidance problem with a supervised terminal objective. To make this problem learnable in high dimensions, we derive an auxiliary Gaussian formulation and prove that solving this randomized problem recovers the optimal deterministic guidance field. This bridge yields a principled continuous-time actor--critic algorithm for learning the guidance module in a fully data-driven manner. Empirically, on AFHQv2 and FFHQ under the pixel EDM pipeline and on ImageNet under the latent EDM2 pipeline, AID consistently improves the quality--speed trade-off over strong fixed-backbone and amortized inpainting baselines across multiple mask types, while adding less than one percent trainable overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。