通过选择特定神经元实现扩散模型持续个性化，高效且不遗忘旧知识。
Continual Personalization for Diffusion Models
- 识别与目标概念相关的神经元，仅微调这些关键神经元。
- 在单/多概念个性化中表现优于现有方法，参数更新极少。
- 无需融合模型，降低存储与计算开销，适合实际部署。
在增量设置下更新扩散模型虽具实际应用价值，但计算成本高。本文提出一种新颖的学习策略——概念神经元选择（CNS），一种简单而有效的持续学习个性化方法。CNS 能唯一识别扩散模型中与目标概念密切相关的神经元。为缓解灾难性遗忘并保持零样本文生图能力，CNS 以增量方式微调这些概念神经元，并共同保留先前概念所学知识。在真实数据集上的评估表明，CNS 仅需极少量参数调整即可达到当前最优性能，在单概念和多概念个性化任务中均超越已有方法。此外，CNS 实现无融合操作，显著减少内存占用和处理时间，适用于持续个性化场景。
原文摘要 · Abstract (English)
Updating diffusion models in an incremental setting would be practical in real-world applications yet computationally challenging. We present a novel learning strategy of Concept Neuron Selection (CNS), a simple yet effective approach to perform personalization in a continual learning scheme. CNS uniquely identifies neurons in diffusion models that are closely related to the target concepts. In order to mitigate catastrophic forgetting problems while preserving zero-shot text-to-image generation ability, CNS finetunes concept neurons in an incremental manner and jointly preserves knowledge learned of previous concepts. Evaluation of real-world datasets demonstrates that CNS achieves state-of-the-art performance with minimal parameter adjustments, outperforming previous methods in both single and multi-concept personalization works. CNS also achieves fusion-free operation, reducing memory storage and processing time for continual personalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。