通过稀疏排斥机制,让扩散模型生成更多样且不重复训练集的图像。
Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
- 在扩散过程加入稀疏排斥项,引导生成路径避开参考图像集。
- 在120万张ImageNet图像上测试,仍能保持高多样性且FID仅小幅上升。
- 无需重新训练,适用于图像保护或提升生成多样性场景。
文本到图像扩散模型的广泛应用引发了可靠性问题,尤其在校准性、公平性和计算效率方面。本文聚焦于部署时的两个核心问题:提示生成的图像多样性不足,以及倾向于重现训练数据中的图像。为此,我们提出一种方法,通过在预训练扩散模型的采样轨迹中加入排斥项,引导生成结果远离参考图像集。排斥项在轨迹可能靠近参考集中图像时被激活,且整体稀疏——多数时间处于关闭状态,尤其在生成末期更为显著。该方法名为SPELL(稀疏排斥),既可使用静态保护图像集,也可动态更新,结合当前批次生成的图像及先前批次结果。实验表明,将SPELL应用于主流扩散模型后,显著提升多样性,同时对FID影响极小,并优于其他近期无训练多样性增强方法。我们还验证了其在包含120万张ImageNet图像的庞大保护集上的有效性。
原文摘要 · Abstract (English)
The adoption of text-to-image diffusion models raises concerns over reliability, drawing scrutiny under the lens of various metrics like calibration, fairness, or compute efficiency. We focus in this work on two issues that arise when deploying these models: a lack of diversity when prompting images, and a tendency to recreate images from the training set. To solve both problems, we propose a method that coaxes the sampled trajectories of pretrained diffusion models to land on images that fall outside of a reference set. We achieve this by adding repellency terms to the diffusion SDE throughout the generation trajectory, which are triggered whenever the path is expected to land too closely to an image in the shielded reference set. Our method is sparse in the sense that these repellency terms are zero and inactive most of the time, and even more so towards the end of the generation trajectory. Our method, named SPELL for sparse repellency, can be used either with a static reference set that contains protected images, or dynamically, by updating the set at each timestep with the expected images concurrently generated within a batch, and with the images of previously generated batches. We show that adding SPELL to popular diffusion models improves their diversity while impacting their FID only marginally, and performs comparatively better than other recent training-free diversity methods. We also demonstrate how SPELL can ensure a shielded generation away from a very large set of protected images by considering all 1.2M images from ImageNet as the protected set.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。