通过早期检测生成错误，提升扩散模型效率
Optimizing Resource Consumption in Diffusion Models through Hallucination Early Detection
- 利用注意力图和预测图像提前判断生成结果是否正确
- 在双对象场景下可节省最多12%的生成时间
- 适合关注生成效率与资源优化的研究者
扩散模型在生成复杂多对象组合时面临挑战，因最终结果高度依赖初始种子，常需多次迭代才能获得满意输出，导致时间和能源浪费。为此，我们提出HEaD（幻觉早期检测）机制，在扩散过程初期快速识别错误生成。该方法结合交叉注意力图与新提出的预测最终图像指标，利用早期生成信息预判最终结果。实验表明，使用HEaD可显著减少计算资源消耗，加速完整图像生成（即所有请求对象均准确呈现），在双对象场景下最高节省12%生成时间，凸显早期检测对生成模型的重要性。
原文摘要 · Abstract (English)
Diffusion models have significantly advanced generative AI, but they encounter difficulties when generating complex combinations of multiple objects. As the final result heavily depends on the initial seed, accurately ensuring the desired output can require multiple iterations of the generation process. This repetition not only leads to a waste of time but also increases energy consumption, echoing the challenges of efficiency and accuracy in complex generative tasks. To tackle this issue, we introduce HEaD (Hallucination Early Detection), a new paradigm designed to swiftly detect incorrect generations at the beginning of the diffusion process. The HEaD pipeline combines cross-attention maps with a new indicator, the Predicted Final Image, to forecast the final outcome by leveraging the information available at early stages of the generation process. We demonstrate that using HEaD saves computational resources and accelerates the generation process to get a complete image, i.e. an image where all requested objects are accurately depicted. Our findings reveal that HEaD can save up to 12% of the generation time on a two objects scenario and underscore the importance of early detection mechanisms in generative models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。