arXiv:2411.10495cs.CV2024-11被引 4

无需训练即可精准控制图像布局,提升生成质量。

Training-Free Layout-to-Image Generation with Marginal Attention Constraints

  • 通过注意力不一致性计算损失,优化生成过程中的潜在特征。
  • 在DrawBench和HRS上优于现有无训练方法,布局准确率更高。
  • 适合需要精确空间控制的图像生成场景,如广告设计、界面原型。

近期许多文本到图像扩散模型在高分辨率图像生成方面表现优异,但在空间构图与物体数量控制上存在不足。为解决此问题,已有研究提出布局到图像(L2I)方法,将布局指令融入预训练模型。然而,现有L2I方法通常需微调参数或训练额外控制模块。本文提出一种无需训练的L2I方法MAC(边际注意力约束生成),避免了额外模块或微调。具体而言,利用文本-视觉交叉注意力特征图量化生成图像与布局指令之间的不一致性,并在扩散逆过程中计算损失以优化潜在特征。为进一步增强空间可控性并减少复杂布局下的语义错误,我们利用自注意力特征图中的像素级相关性对齐交叉注意力图,并结合三种由边界注意力约束的损失函数更新潜在特征。在多个预训练扩散模型上的实验表明,该方法在DrawBench和HRS基准测试中,无论定量还是定性评价均优于现有无训练方法。

原文摘要 · Abstract (English)

Recently, many text-to-image diffusion models have excelled at generating high-resolution images from text but struggle with precise control over spatial composition and object counting. To address these challenges, prior works have developed layout-to-image (L2I) approaches that incorporate layout instructions into text-to-image models. However, existing L2I methods typically require fine-tuning of pre-trained parameters or training additional control modules for diffusion models. In this work, we propose a training-free L2I approach, MAC (Marginal Attention Constrained Generation), which eliminates the need for additional modules or fine-tuning. Specifically, we use text-visual cross-attention feature maps to quantify inconsistencies between the layout of the generated images and the provided instructions, and then compute loss functions to optimize latent features during the diffusion reverse process. To enhance spatial controllability and mitigate semantic failures under complex layout instructions, we leverage pixel-to-pixel correlations in self-attention feature maps to align cross-attention maps and combine three loss functions constrained by boundary attention to update latent features. Comprehensive experimental results on both L2I and non-L2I pretrained diffusion models demonstrate that our method outperforms existing training-free L2I techniques, both quantitatively and qualitatively, in terms of image composition on the DrawBench and HRS benchmarks.

图像生成扩散模型布局控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。