提出GUARD框架,精准消除文本到图像模型的训练数据记忆问题。
You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models
- 通过吸引-排斥动力学动态调整生成过程,避开原始训练图。
- 在两种架构上均显著降低原文复现率,且保持图像质量不下降。
- 适合关注隐私与版权风险的研究者或应用开发者。
生成模型会“记忆”训练数据,导致生成与训练样本几乎相同的图像,引发隐私泄露或版权争议。本文提出引导性吸引-排斥动力学(GUARD),一种用于文本到图像扩散模型的记忆缓解新框架。该框架在去噪过程中引导生成结果远离原始训练图像,同时保持与提示词一致,从而避免复现训练数据,且不影响生成质量。我们设计了一种具体实现:基于新颖统计机制自动识别需减弱跨注意力的提示位置,并在这些位置进行注意力衰减。所提出的GUARD是一种针对提示动态、精准干预的推理时方法,在两种模型架构上对原文复现和模板记忆均表现出最强鲁棒性,显著优于现有方法,同时在图像质量上表现相当或更优。
原文摘要 · Abstract (English)
Generative models have been shown to "memorize" certain training data, leading to verbatim or near-verbatim generating images, which may cause privacy concerns or copyright infringement. We introduce Guidance Using Attractive-Repulsive Dynamics (GUARD), a novel framework for memorization mitigation in text-to-image diffusion models. GUARD adjusts the image denoising process to guide the generation away from an original training image and towards one that is distinct from training data while remaining aligned with the prompt, guarding against reproducing training data, without hurting image generation quality. We propose a concrete instantiation of this framework, where the positive target that we steer towards is given by a novel method for (cross) attention attenuation based on (i) a novel statistical mechanism that automatically identifies the prompt positions where cross attention must be attenuated and (ii) attenuating cross-attention in these per-prompt locations. The resulting GUARD offers a surgical, dynamic per-prompt inference-time approach that, we find, is by far the most robust method in terms of consistently producing state-of-the-art results for memorization mitigation across two architectures and for both verbatim and template memorization, while also improving upon or yielding comparable results in terms of image quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。