arXiv:2604.04924cs.CVcs.AI2026-04

无需微调,直接学提示词就能让扩散模型自动修复图像

Your Pre-trained Diffusion Model Secretly Knows Restoration

  • 在文本编码器输出处学习提示嵌入,激活预训练模型的修复能力
  • 新方法在多种退化场景下表现接近主流模型,且无需额外模块
  • 适合想快速部署修复功能而不想调参的研究者和开发者

预训练扩散模型在全场景图像修复(AiOR)中展现出优异的感知质量和泛化能力。然而,现有方法主要依赖微调或Control-Net模块来利用其先验知识。本文发现,这些模型本身具备内在修复行为,可通过直接学习文本编码器输出处的提示嵌入来激活。值得注意的是,这种能力难以通过传统文本提示或文本嵌入优化获得。此外,朴素提示学习因前向加噪过程与反向采样轨迹不一致而出现不稳定问题。为此,我们采用扩散桥框架,在训练与推理间保持一致的去噪路径。基于此,我们在预训练的WAN视频模型和FLUX图像模型上引入轻量级可学习提示,将其转化为高性能修复模型。大量实验表明,该方法在多种退化类型下表现优异,兼具良好泛化性,且无需微调或专用控制模块。

原文摘要 · Abstract (English)

Pre-trained diffusion models have enabled significant advancements in All-in-One Restoration (AiOR), offering improved perceptual quality and generalization. However, diffusion-based restoration methods primarily rely on fine-tuning or Control-Net style modules to leverage the pre-trained diffusion model's priors for AiOR. In this work, we show that these pre-trained diffusion models inherently possess restoration behavior, which can be unlocked by directly learning prompt embeddings at the output of the text encoder. Interestingly, this behavior is largely inaccessible through text prompts and text-token embedding optimization. Furthermore, we observe that naive prompt learning is unstable because the forward noising process using degraded images is misaligned with the reverse sampling trajectory. To resolve this, we train prompts within a diffusion bridge formulation that aligns training and inference dynamics, enforcing a coherent denoising path from noisy degraded states to clean images. Building on these insights, we introduce our lightweight learned prompts on the pre-trained WAN video model and FLUX image models, converting them into high-performing restoration models. Extensive experiments demonstrate that our approach achieves competitive performance and generalization across diverse degradations, while avoiding fine-tuning and restoration-specific control modules.

图像修复扩散模型提示学习零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。