arXiv:2503.06506cs.CVcs.LG2025-03被引 5

不训练模型,直接优化文本约束,提升图像生成的细节准确性。

Fine-Grained Alignment and Noise Refinement for Compositional Text-to-Image Generation

  • 用文本约束设计统一损失函数,精准控制实体、属性和空间关系。
  • 人评提升24%,空间关系准确率提高25%,噪声精修最多增效5%。
  • 适合需要高精度图文对齐的生成场景,如复杂描述生成。

近年来,文本到图像生成模型取得了显著进展;然而,准确捕捉文本提示中的细微细节——如实体缺失、属性绑定错误和关系错误——仍是巨大挑战。为此,我们提出一种无需训练的方法,通过引入定制化目标直接应对这些挑战。与强制刚性布局的现有方法不同,本方法仅根据文本提取约束条件,不添加额外结构,实现更灵活的场景构图。这些约束以损失形式呈现:实体缺失、实体混淆、属性绑定和空间关系,并整合进统一损失中,应用于首次生成阶段。此外,我们设计了反馈驱动的初始噪声精修系统:通过验证器评估生成图像,识别不一致并提供修正反馈。基于此反馈,我们的精修方法首先针对未满足的约束,优化由初始噪声引起的错误注意力图,通过调整相关约束的可选损失;随后重新应用统一损失,进入第二轮生成。实验表明,仅依赖所提目标函数,该方法显著提升组合性表现,人评提升24%,空间关系准确率提高25%。细粒度噪声精修有效提升性能,最高达5%。代码已公开于 https://github.com/hadi-hosseini/noise-refinement。

原文摘要 · Abstract (English)

Text-to-image generative models have made significant advancements in recent years; however, accurately capturing intricate details in textual prompts-such as entity missing, attribute binding errors, and incorrect relationships remains a formidable challenge. In response, we present an innovative, training-free method that directly addresses these challenges by incorporating tailored objectives to account for textual constraints. Unlike layout-based approaches that enforce rigid structures and limit diversity, our proposed approach offers a more flexible arrangement of the scene by imposing just the extracted constraints from the text, without any unnecessary additions. These constraints are formulated as losses-entity missing, entity mixing, attribute binding, and spatial relationships-integrated into a unified loss that is applied in the first generation stage. Furthermore, we introduce a feedback-driven system for fine-grained initial noise refinement. This system integrates a verifier that evaluates the generated image, identifies inconsistencies, and provides corrective feedback. Leveraging this feedback, our refinement method first targets the unmet constraints by refining the faulty attention maps caused by initial noise, through the optimization of selective losses associated with these constraints. Subsequently, our unified loss function is reapplied to proceed the second generation phase. Experimental results demonstrate that our method, relying solely on our proposed objective functions, significantly enhances compositionality, achieving a 24% improvement in human evaluation and a 25% gain in spatial relationships. Furthermore, our fine-grained noise refinement proves effective, boosting performance by up to 5%. Code is available at \href{https://github.com/hadi-hosseini/noise-refinement}{https://github.com/hadi-hosseini/noise-refinement}.

文本生成图像生成细粒度对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。