让扩散模型在微调时不学坏概念，靠一句话就能控制。
Coffee: Controllable Diffusion Fine-tuning
- 用语言指定不该学的坏概念，通过阻止提示词与坏概念对齐来实现控制。
- 实验表明能有效防止模型学习指定的不良概念，效果优于现有方法。
- 无需额外训练，改描述即可灵活调整要避开的概念，适合安全微调场景。
文本到图像的扩散模型可通过少量用户数据进行定制化微调，但如何在微调中避免学习数据中隐含的不良概念，以及防止这些概念与用户提示纠缠，仍是未解难题。这对偏见缓解、防止恶意适应、属性解耦和通用策略微调至关重要。本文提出 Coffee，通过语言指定需规避的不良概念，以正则化微调过程。核心思路是阻止用户提示的嵌入向量与不良概念对齐。Coffee无需额外训练，仅通过修改文本描述即可灵活调整需规避的概念。在包含用户提示与不良概念关联图像的数据集上进行微调实验，结果表明 Coffee 能有效阻止模型学习指定不良概念，性能超越现有方法。代码将在论文接受后发布。
原文摘要 · Abstract (English)
Text-to-image diffusion models can generate diverse content with flexible prompts, which makes them well-suited for customization through fine-tuning with a small amount of user-provided data. However, controllable fine-tuning that prevents models from learning undesired concepts present in the fine-tuning data, and from entangling those concepts with user prompts, remains an open challenge. It is crucial for downstream tasks like bias mitigation, preventing malicious adaptation, attribute disentanglement, and generalizable fine-tuning of diffusion policy. We propose Coffee that allows using language to specify undesired concepts to regularize the adaptation process. The crux of our method lies in keeping the embeddings of the user prompt from aligning with undesired concepts. Crucially, Coffee requires no additional training and enables flexible modification of undesired concepts by modifying textual descriptions. We evaluate Coffee by fine-tuning on images associated with user prompts paired with undesired concepts. Experimental results demonstrate that Coffee can prevent text-to-image models from learning specified undesired concepts during fine-tuning and outperforms existing methods. Code will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。