让文生图模型持续学习新概念，不遗忘旧概念。
How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?
- 提出增量式学习框架，通过知识整合缓解遗忘。
- 新旧概念生成均保持高保真，无显著性能下降。
- 适合需要长期更新个性化图像生成需求的用户。
个性化扩散模型（CDMs）因其强大的生成能力受到广泛关注，但现有方法通常假设个性化概念固定不变，无法随时间更新。在持续学习新概念时，普遍存在灾难性遗忘和旧概念忽略问题。为此，本文提出概念增量式文生图模型（CIDM），可实现新定制任务的持续学习而不遗忘旧概念。具体而言，通过设计概念巩固损失与弹性权重聚合模块，探索任务特异性与共享知识，并在推理时基于贡献度聚合旧概念的低秩权重。此外，为缓解概念忽略，引入上下文可控合成策略，利用表达性区域特征与噪声估计，根据用户条件控制生成图像的上下文。实验表明，该方法优于现有定制化扩散模型。源代码已开源：https://github.com/JiahuaDong/CIFC。
原文摘要 · Abstract (English)
Custom diffusion models (CDMs) have attracted widespread attention due to their astonishing generative ability for personalized concepts. However, most existing CDMs unreasonably assume that personalized concepts are fixed and cannot change over time. Moreover, they heavily suffer from catastrophic forgetting and concept neglect on old personalized concepts when continually learning a series of new concepts. To address these challenges, we propose a novel Concept-Incremental text-to-image Diffusion Model (CIDM), which can resolve catastrophic forgetting and concept neglect to learn new customization tasks in a concept-incremental manner. Specifically, to surmount the catastrophic forgetting of old concepts, we develop a concept consolidation loss and an elastic weight aggregation module. They can explore task-specific and task-shared knowledge during training, and aggregate all low-rank weights of old concepts based on their contributions during inference. Moreover, in order to address concept neglect, we devise a context-controllable synthesis strategy that leverages expressive region features and noise estimation to control the contexts of generated images according to user conditions. Experiments validate that our CIDM surpasses existing custom diffusion models. The source codes are available at https://github.com/JiahuaDong/CIFC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。