揭示扩散模型中前景背景的局部记忆现象，发现现有方法无法根除局部记忆。
Demystifying Foreground-Background Memorization in Diffusion Models
- 提出基于分割的FB-Mem度量方法，量化生成图像中的局部记忆区域。
- 发现单个提示生成结果可能关联多个训练图像，记忆模式远超一对一匹配。
- 证明现有剪枝等方法无法消除前景区域的记忆，需改进为聚类驱动的强化方案。
扩散模型（DMs）会记忆训练图像，并在生成时复现近似副本。现有检测方法仅能识别完整记忆，却无法衡量小区域内的部分记忆，也难以捕捉超出特定提示-图像对的记忆模式。为此，我们提出前景-背景记忆（FB-Mem）度量方法，通过分割技术分类并量化生成图像中的记忆区域。研究发现：（1）单个提示生成的结果可能关联多个相似训练图像，揭示了超越一对一对应关系的复杂记忆模式；（2）现有的模型级缓解方法（如神经元去激活与剪枝）无法消除局部记忆，尤其在前景区域仍持续存在。本工作建立了一种有效的扩散模型记忆测量框架，揭示了当前缓解手段的不足，并提出一种基于聚类的更强缓解策略。
原文摘要 · Abstract (English)
Diffusion models (DMs) memorize training images and can reproduce near-duplicates during generation. Current detection methods identify verbatim memorization but fail to capture two critical aspects: quantifying partial memorization occurring in small image regions, and memorization patterns beyond specific prompt-image pairs. To address these limitations, we propose Foreground Background Memorization (FB-Mem), a novel segmentation-based metric that classifies and quantifies memorized regions within generated images. Our method reveals that memorization is more pervasive than previously understood: (1) individual generations from single prompts may be linked to clusters of similar training images, revealing complex memorization patterns that extend beyond one-to-one correspondences; and (2) existing model-level mitigation methods, such as neuron deactivation and pruning, fail to eliminate local memorization, which persists particularly in foreground regions. Our work establishes an effective framework for measuring memorization in diffusion models, demonstrates the inadequacy of current mitigation approaches, and proposes a stronger mitigation method using a clustering approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。