提出DuMo网络，精准擦除图像中的不当概念，几乎不损伤其他内容。
DuMo: Dual Encoder Modulation Network for Precise Concept Erasure
- 通过修改U-Net跳接特征，仅针对图像细节（高频）进行概念擦除。
- 冻结主干网络参数，引入原始跳接特征作为先验知识，减少对非目标内容影响。
- 设计时间-层级调制机制，自动平衡不同阶段的擦除强度与生成能力。
文本到图像模型强大的生成能力引发了关于不适宜内容（NSFW）和潜在版权侵权的安全担忧。以往方法通过消除不当概念来保护模型，但这类方法会改变主干网络参数，显著影响图像结构（低频）成分，削弱模型保留非目标概念的能力。本文提出双编码调制网络DuMo，实现对不当概念的精准擦除，同时最大限度减少对非目标概念的影响。不同于以往方法，DuMo采用带有先验知识的擦除模块（EPR），仅修改U-Net跳接特征，主要作用于图像细节（高频）部分。为最小化擦除过程中对非目标概念的破坏，主干U-Net参数保持冻结，并将原始跳接特征作为先验引入擦除过程。此外，观察到EPR在不同时间步和层上对图像结构与细节表现出不同的擦除偏好。为此，我们提出新颖的时间-层级调制过程（TLMO），动态调节EPR输出的擦除尺度，实现擦除效果与生成能力的自动平衡。实验表明，该方法在显性内容擦除、卡通概念移除和艺术风格擦除任务上均达到当前最优性能。代码已公开于https://github.com/Maplebb/DuMo。
原文摘要 · Abstract (English)
The exceptional generative capability of text-to-image models has raised substantial safety concerns regarding the generation of Not-Safe-For-Work (NSFW) content and potential copyright infringement. To address these concerns, previous methods safeguard the models by eliminating inappropriate concepts. Nonetheless, these models alter the parameters of the backbone network and exert considerable influences on the structural (low-frequency) components of the image, which undermines the model's ability to retain non-target concepts. In this work, we propose our Dual encoder Modulation network (DuMo), which achieves precise erasure of inappropriate target concepts with minimum impairment to non-target concepts. In contrast to previous methods, DuMo employs the Eraser with PRior Knowledge (EPR) module which modifies the skip connection features of the U-NET and primarily achieves concept erasure on details (high-frequency) components of the image. To minimize the damage to non-target concepts during erasure, the parameters of the backbone U-NET are frozen and the prior knowledge from the original skip connection features is introduced to the erasure process. Meanwhile, the phenomenon is observed that distinct erasing preferences for the image structure and details are demonstrated by the EPR at different timesteps and layers. Therefore, we adopt a novel Time-Layer MOdulation process (TLMO) that adjusts the erasure scale of EPR module's outputs across different layers and timesteps, automatically balancing the erasure effects and model's generative ability. Our method achieves state-of-the-art performance on Explicit Content Erasure, Cartoon Concept Removal and Artistic Style Erasure, clearly outperforming alternative methods. Code is available at https://github.com/Maplebb/DuMo
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。