提前检测扩散模型生成中的幻觉,减少无效计算。
Hallucination Early Detection in Diffusion Models

- 利用交叉注意力与预测图像早期判断生成是否出错。
- 四物体场景下完整生成率提升6-8%,生成时间缩短32%。
- 支持对象位置与空间关系验证,适合复杂多物场景生成。
文本到图像生成在扩散模型推动下已实现高度逼真。然而,当需生成多个物体时,扩散模型常出现遗漏实体的幻觉问题。现有方法多聚焦于优化隐空间表示,却忽视初始种子的重要性。多次尝试不同种子虽可改善结果,但代价高昂。为此,我们提出HEaD+(幻觉早期检测+),通过融合交叉注意力图、文本信息与预测最终图像,在扩散过程早期判断是否继续当前生成或换种子重启,以较少迭代达成更完整生成。该框架在新构建的InsideGen数据集(45,000张含最多七个物体的图像)上训练。实验表明,结合现有模型使用时,四物体场景下完整生成概率提升6%-8%;为获得完整图像,生成时间最多减少32%。此外,我们引入定位模块,在中间步骤预测物体中心位置并验证成对空间关系(用户指定时),通过对象存在性门控进一步提升关系一致性表现。
原文摘要 · Abstract (English)
Text-to-Image generation has seen significant advancements in output realism with the advent of diffusion models. However, diffusion models encounter difficulties when tasked with generating multiple objects, frequently resulting in hallucinations where certain entities are omitted. While existing solutions typically focus on optimizing latent representations within diffusion models, the relevance of the initial generation seed is typically underestimated. While using various seeds in multiple iterations can improve results, this method also significantly increases time and energy costs. To address this challenge, we introduce HEaD+ (Hallucination Early Detection +), a novel approach designed to identify incorrect generations early in the diffusion process. The HEaD+ framework integrates cross-attention maps and textual information with a novel input, the Predicted Final Image. The objective is to assess whether to proceed with the current generation or restart it with a different seed, thereby exploring multiple-generation seeds while conserving time. HEaD+ is trained on the newly created InsideGen dataset of 45,000 generated images, each containing prompts with up to seven objects. Our findings demonstrate a 6-8% increase in the likelihood of achieving a complete generation (i.e., an image accurately representing all specified subjects) with four objects when applying HEaD+ alongside existing models. Additionally, HEaD+ reduces generation times by up to 32% when aiming for a complete image, enhancing the efficiency of generating complete and accurate object representations relative to leading models. Moreover, we propose an integrated localization module that predicts object centroid positions and verifies pairwise spatial relations (if requested by the users) at an intermediate timestep, gating generation together with object presence to further improve relation-consistent outcomes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。