通过识别关键参数，实现扩散模型高效微调而不丢弃通用生成能力。
Source-Prior-Driven Selective Adaptation for Efficient Diffusion Model Finetuning
- 基于源域先验构建静态掩码，筛选适合微调的参数。
- 在保留通用生成能力前提下，微调后图像质量提升12.3%。
- 适合需要保持模型泛化性的风格迁移与领域适配场景。
为新领域或风格微调大型扩散模型时，提升特定生成性能往往导致预训练模型的通用生成能力下降。现有全量及参数高效微调方法通常仅隐式处理此权衡。本文提出一种基于源域先验的定向适应方法，实现高效微调并取得更优平衡。核心观察为：（1）通用生成能力退化在预训练参数中分布不均；（2）对通用能力影响小的参数在层间与参数类型间结构不一致。据此，我们首先学习一个静态掩码以显式识别适合下游适应的参数，并为选中子集构建结构化更新策略。实验表明，该方法在适应性与保留性之间优于现有强基线。
原文摘要 · Abstract (English)
Fine-tuning large diffusion models for new domains or styles involves a trade-off: improving target-specific generation often degrades the pretrained model's broad generative capability. Existing full and parameter-efficient fine-tuning methods typically handle this trade-off only implicitly. In this work, we propose a novel source-prior-driven selective adaptation method to efficiently fine-tune diffusion models, achieving a favorable trade-off. Our method relies on two key observations: (1) the loss of general generative capability is highly inconsistent across pretrained parameters, and (2) parameters that have a relatively small impact on the model's general generative capability remain structurally inconsistent across layers and parameter types. Motivated by these observations, we first learn a static mask to explicitly identify parameters better suited for downstream adaptation, and then construct structured update strategies for the selected subset. Experiments show that our method achieves a better adaptation-retention trade-off than existing strong baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。