无需重训练,通过调整文本条件令牌实现对图像生成中敏感概念的高效擦除。
CE-SDWV: Effective and Efficient Concept Erasure for Text-to-Image Diffusion Models via a Semantic-Driven Word Vocabulary
- 在文本语义空间中构建目标概念词汇表,增强对敏感概念的表示。
- 采用自适应语义组件抑制策略,从文本条件中移除目标概念信息。
- 适用于需快速移除特定概念的隐私安全场景,如去除非安全内容。
大规模文本到图像(T2I)扩散模型在生成各类概念方面表现卓越。然而,在实际应用中,隐私与安全限制使得生成不适宜内容(如NSFW概念、受版权保护图像)的能力不可接受。因此,针对T2I扩散模型的概念擦除任务受到广泛关注,亟需一种高效且有效的方法。为此,我们提出CE-SDWV框架,仅通过调整文本条件令牌来实现对目标概念(如NSFW概念)的擦除,无需重新训练原始模型权重。具体而言,该框架首先构建与目标概念相关的词典,增强其在文本语义空间中的表达;随后采用自适应语义组件抑制策略,消除文本条件令牌中的相关语义信息。为进一步使优化后的文本条件适配原始图像语义空间,我们设计了端到端梯度正交令牌优化策略。在I2P和UnlearnCanvas基准上的大量实验验证了方法的有效性与效率。代码已公开于https://github.com/TtuHamg/CE-SDWV。
原文摘要 · Abstract (English)
Large-scale text-to-image (T2I) diffusion models have achieved remarkable generative performance about various concepts. With the limitation of privacy and safety in practice, the generative capability concerning NSFW (Not Safe For Work) concepts is undesirable, e.g., producing sexually explicit photos, and licensed images. The concept erasure task for T2I diffusion models has attracted considerable attention and requires an effective and efficient method. To achieve this goal, we propose a CE-SDWV framework, which removes the target concepts (e.g., NSFW concepts) of T2I diffusion models in the text semantic space by only adjusting the text condition tokens and does not need to re-train the original T2I diffusion model's weights. Specifically, our framework first builds a target concept-related word vocabulary to enhance the representation of the target concepts within the text semantic space, and then utilizes an adaptive semantic component suppression strategy to ablate the target concept-related semantic information in the text condition tokens. To further adapt the above text condition tokens to the original image semantic space, we propose an end-to-end gradient-orthogonal token optimization strategy. Extensive experiments on I2P and UnlearnCanvas benchmarks demonstrate the effectiveness and efficiency of our method. Code is available at https://github.com/TtuHamg/CE-SDWV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。