arXiv:2504.11850cs.CV2025-04被引 3

让扩散模型删除特定概念,生成更安全的图像。

ACE: Attentional Concept Erasure in Diffusion Models

  • 通过注意力机制识别并清除特定概念的特征方向。
  • 在多个数据集上实现顶尖去概念效果,支持数十个概念批量处理。
  • 仅需数秒微调,保留其他内容生成能力,适合实际部署。

大型文本到图像扩散模型虽具备强大生成能力,但因在互联网规模数据上无差别训练,可能习得有害、版权或不希望出现的内容。本文提出注意力概念擦除(ACE)方法,旨在从预训练模型中移除指定概念,使提示该概念或相关词汇不再生成其图像,同时保持对其他内容的生成能力。理论上,将目标概念的条件分布对齐至中性分布。方法通过门控低秩适配,在交叉注意力模块中识别并归零特定概念的潜在方向,再经对抗增强微调确保彻底擦除概念及其同义词。实验证明,ACE在物体类别、名人面部、敏感内容和艺术风格等多个基准上均达到当前最优去概念效果,兼具广度(覆盖相关术语)与精度(保留无关内容),可扩展至数十个概念,且每概念适应仅需数秒。代码将公开,助力扩散模型更安全落地。

原文摘要 · Abstract (English)

Large text-to-image diffusion models have demonstrated remarkable image synthesis capabilities, but their indiscriminate training on Internet-scale data has led to learned concepts that enable harmful, copyrighted, or otherwise undesirable content generation. We address the task of concept erasure in diffusion models, i.e., removing a specified concept from a pre-trained model such that prompting the concept (or related synonyms) no longer yields its depiction, while preserving the model's ability to generate other content. We propose a novel method, Attentional Concept Erasure (ACE), that integrates a closed-form attention manipulation with lightweight fine-tuning. Theoretically, we formulate concept erasure as aligning the model's conditional distribution on the target concept with a neutral distribution. Our approach identifies and nullifies concept-specific latent directions in the cross-attention modules via a gated low-rank adaptation, followed by adversarially augmented fine-tuning to ensure thorough erasure of the concept and its synonyms. Empirically, we demonstrate on multiple benchmarks, including object classes, celebrity faces, explicit content, and artistic styles, that ACE achieves state-of-the-art concept removal efficacy and robustness. Compared to prior methods, ACE better balances generality (erasing concept and related terms) and specificity (preserving unrelated content), scales to dozens of concepts, and is efficient, requiring only a few seconds of adaptation per concept. We will release our code to facilitate safer deployment of diffusion models.

扩散模型概念擦除安全性微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。