通过细粒度概率加噪，有效降低扩散模型的数据复现问题。
FPAN: Mitigating Replication in Diffusion Models through the Fine-Grained Probabilistic Addition of Noise to Token Embeddings
- 对文本嵌入进行概率性加噪，噪声量随词元动态调整。
- 相比基线模型，平均减少28.78%的数据复现，优于此前方法26.51%。
- 可与现有方法结合使用，提升隐私保护效果，不影响图像质量。
扩散模型在生成高质量图像方面表现出巨大潜力，但其容易复制训练数据，引发隐私担忧，尤其当训练数据包含敏感信息时。现有缓解策略主要集中在减少图像重复、修改交叉注意力机制或改变去噪主干结构。近期研究发现，在文本嵌入中添加固定小量噪声可部分降低复现率。本文首先分析不同噪声强度的影响，提出一种细粒度概率加噪方法（FPAN），对词元嵌入按概率注入更大噪声。大量实验表明，相较于无噪声基线模型，FPAN平均降低28.78%的复现率，优于先前固定噪声方法26.51%；当与已有方法结合时，复现率进一步下降16.82%，且图像质量保持相当甚至更好。
原文摘要 · Abstract (English)
Diffusion models have demonstrated remarkable potential in generating high-quality images. However, their tendency to replicate training data raises serious privacy concerns, particularly when the training datasets contain sensitive or private information. Existing mitigation strategies primarily focus on reducing image duplication, modifying the cross-attention mechanism, and altering the denoising backbone architecture of diffusion models. Moreover, recent work has shown that adding a consistent small amount of noise to text embeddings can reduce replication to some degree. In this work, we begin by analyzing the impact of adding varying amounts of noise. Based on our analysis, we propose a fine-grained noise injection technique that probabilistically adds a larger amount of noise to token embeddings. We refer to our method as Fine-grained Probabilistic Addition of Noise (FPAN). Through our extensive experiments, we show that our proposed FPAN can reduce replication by an average of 28.78% compared to the baseline diffusion model without significantly impacting image quality, and outperforms the prior consistent-magnitude-noise-addition approach by 26.51%. Moreover, when combined with other existing mitigation methods, our FPAN approach can further reduce replication by up to 16.82% with similar, if not improved, image quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。