用大模型引导噪声优化，让图文生成更精准。
Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis
- 利用视觉语言模型分析提示语，优化生成初始噪声。
- 在多个扩散模型上提升语义对齐，保持图像分布一致。
- 理论分析更新条件，适合需要高精度图文对齐的研究者。
扩散模型在生成逼真图像方面取得显著进展，但在精确对齐输入提示语的语义方面仍面临挑战。优化初始噪声隐变量是一种比修改模型架构或提示工程更高效的方法。最新方法InitNo通过注意力图优化初始噪声，但其信息捕捉有限，且易收敛至起始点附近的局部最优解。为此，本文提出利用大视觉语言模型（LVLM）的语言理解能力，指导初始噪声隐变量的优化，并引入噪声扩散过程，在保持分布一致性的同时生成语义忠实的图像。此外，我们提供了更新机制改善语义忠实度的理论分析。实验表明，该框架在多种扩散模型上均能持续提升语义对齐效果。代码已开源：https://github.com/Bomingmiao/NoiseDiffusion。
原文摘要 · Abstract (English)
Diffusion models have achieved impressive success in generating photorealistic images, but challenges remain in ensuring precise semantic alignment with input prompts. Optimizing the initial noisy latent offers a more efficient alternative to modifying model architectures or prompt engineering for improving semantic alignment. A latest approach, InitNo, refines the initial noisy latent by leveraging attention maps; however, these maps capture only limited information, and the effectiveness of InitNo is highly dependent on the initial starting point, as it tends to converge on a local optimum near this point. To this end, this paper proposes leveraging the language comprehension capabilities of large vision-language models (LVLMs) to guide the optimization of the initial noisy latent, and introduces the Noise Diffusion process, which updates the noisy latent to generate semantically faithful images while preserving distribution consistency. Furthermore, we provide a theoretical analysis of the condition under which the update improves semantic faithfulness. Experimental results demonstrate the effectiveness and adaptability of our framework, consistently enhancing semantic alignment across various diffusion models. The code is available at https://github.com/Bomingmiao/NoiseDiffusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。