用低秩适配快速移除图像模型中的特定概念,50秒内仅需4-5张图。
Few-Shot Concept Unlearning with Low Rank Adaptation
- 通过文本编码器最后一层梯度更新实现概念移除。
- 仅用4-5张图像,平均50秒完成概念去学习。
- 适合需要快速删除敏感内容的生成模型应用。
图像生成模型常会生成涉及隐私或版权问题的敏感内容。机器去学习旨在移除特定数据子集对模型的影响,尤其在生成模型中需消除对某一概念的生成能力。传统重训练耗时可达数天,效率低下。本文提出一种基于低秩适配与文本反转的快速算法,通过加权损失函数在稳定扩散模型的文本编码器最后几层进行反向传播,更新权重以消除概念在文本-图像嵌入空间中的影响,使模型在提示下不再生成该概念相关图像。实验基于潜变量扩散模型(如Stable Diffusion v2),使用4-5张图像即可在平均50秒内完成概念去学习。
原文摘要 · Abstract (English)
Image Generation models are a trending topic nowadays, with many people utilizing Artificial Intelligence models in order to generate images. There are many such models which, given a prompt of a text, will generate an image which depicts said prompt. There are many image generation models, such as Latent Diffusion Models, Denoising Diffusion Probabilistic Models, Generative Adversarial Networks and many more. When generating images, these models can generate sensitive image data, which can be threatening to privacy or may violate copyright laws of private entities. Machine unlearning aims at removing the influence of specific data subsets from the trained models and in the case of image generation models, remove the influence of a concept such that the model is unable to generate said images of the concept when prompted. Conventional retraining of the model can take upto days, hence fast algorithms are the need of the hour. In this paper we propose an algorithm that aims to remove the influence of concepts in diffusion models through updating the gradients of the final layers of the text encoders. Using a weighted loss function, we utilize backpropagation in order to update the weights of the final layers of the Text Encoder componet of the Stable Diffusion Model, removing influence of the concept from the text-image embedding space, such that when prompted, the result is an image not containing the concept. The weighted loss function makes use of Textual Inversion and Low-Rank Adaptation.We perform our experiments on Latent Diffusion Models, namely the Stable Diffusion v2 model, with an average concept unlearning runtime of 50 seconds using 4-5 images.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。