微调生成模型会引发大范围语义与外观漂移，影响原有能力。
Assessing Open-world Forgetting in Generative Image Model Customization
- 提出开放世界遗忘概念，系统分析微调导致的语义与外观变化。
- 微调后旧概念识别准确率下降最高达60%，纹理颜色分布显著改变。
- 引入功能正则化策略，有效保留原模型能力并适配新类。
扩散模型在图像生成方面取得显著进展，但通过新增类别进行定制时，常引发不可预见的后果，损害模型可靠性。本文首次系统研究扩散模型中的开放世界遗忘现象，聚焦表征的语义与外观漂移。通过零样本分类实验发现，即使轻微的模型调整也会导致显著的语义漂移，影响远超新增概念的原有知识，使旧概念识别准确率最高下降60%。对外观漂移的分析显示生成内容的纹理和颜色分布发生大幅改变。为此，我们提出一种功能正则化策略，在保留原始能力的同时有效整合新概念。在多个数据集和评估指标上的大量实验表明，该方法显著降低了语义与外观漂移。研究强调了未来模型定制与微调方法中需考虑开放世界遗忘的重要性。
原文摘要 · Abstract (English)
Recent advances in diffusion models have significantly enhanced image generation capabilities. However, customizing these models with new classes often leads to unintended consequences that compromise their reliability. We introduce the concept of open-world forgetting to characterize the vast scope of these unintended alterations. Our work presents the first systematic investigation into open-world forgetting in diffusion models, focusing on semantic and appearance drift of representations. Using zero-shot classification, we demonstrate that even minor model adaptations can lead to significant semantic drift affecting areas far beyond newly introduced concepts, with accuracy drops of up to 60% on previously learned concepts. Our analysis of appearance drift reveals substantial changes in texture and color distributions of generated content. To address these issues, we propose a functional regularization strategy that effectively preserves original capabilities while accommodating new concepts. Through extensive experiments across multiple datasets and evaluation metrics, we demonstrate that our approach significantly reduces both semantic and appearance drift. Our study highlights the importance of considering open-world forgetting in future research on model customization and finetuning methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。