通过动态调节注意力机制，有效减少扩散模型生成图像中的幻觉问题。
Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulation
- 在去噪早期动态调整注意力分布，抑制错误特征放大。
- 在双手数据集上使幻觉图像比例下降12.9%，FID分数提升20.8%。
- 适合关注图像真实性与生成可靠性研究的开发者和工程师。
扩散模型虽能生成逼真图像，但常出现与训练数据分布不符的幻觉——即不真实或错误的细节。本文提出自适应注意力调制（AAM），通过分析并调控扩散模型中的自注意力机制来缓解幻觉。我们假设：在去噪初期，自注意力可能无意中放大或抑制某些特征，从而引发幻觉。为此，AAM在自注意力层的softmax操作中引入温度缩放机制，动态调节推理过程中的注意力分布；同时采用掩码扰动技术，干扰早期噪声，防止其在后续阶段演化为幻觉。大量实验表明，AAM显著降低幻觉现象，提升生成图像的保真度与可靠性。例如，在Hands数据集上，该方法使幻觉图像比例下降12.9%（绝对值），FID分数改善20.8%。
原文摘要 · Abstract (English)
Diffusion models, while increasingly adept at generating realistic images, are notably hindered by hallucinations -- unrealistic or incorrect features inconsistent with the trained data distribution. In this work, we propose Adaptive Attention Modulation (AAM), a novel approach to mitigate hallucinations by analyzing and modulating the self-attention mechanism in diffusion models. We hypothesize that self-attention during early denoising steps may inadvertently amplify or suppress features, contributing to hallucinations. To counter this, AAM introduces a temperature scaling mechanism within the softmax operation of the self-attention layers, dynamically modulating the attention distribution during inference. Additionally, AAM employs a masked perturbation technique to disrupt early-stage noise that may otherwise propagate into later stages as hallucinations. Extensive experiments demonstrate that AAM effectively reduces hallucinatory artifacts, enhancing both the fidelity and reliability of generated images. For instance, the proposed approach improves the FID score by 20.8% and reduces the percentage of hallucinated images by 12.9% (in absolute terms) on the Hands dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。