arXiv:2411.16783cs.CV2024-11被引 8

通过优化注意力关系,让图文生成更准地匹配提示词内容

CoCoNO: Attention Contrast-and-Complete for Initial Noise Optimization in Text-to-Image Synthesis

  • 利用自注意力与交叉注意力的互补信息优化初始噪声
  • 在COCO、ImageNet等数据集上显著提升图文对齐效果
  • 无需重训练模型,适合希望提升生成精度的研究者

尽管文本到图像生成模型取得进展,但实现语义准确的图像仍是挑战。现有初始隐空间优化方法存在两个关键问题:(a) 注意力忽略,即输入提示中的某些主体虽在交叉注意力中响应高,却未在自注意力图中对应特定区域;(b) 注意力干扰,即不同主体的交叉与自注意力重叠导致生成图像属性混淆。为此,我们提出CoCoNO,通过联合利用自注意力与交叉注意力的互补信息优化初始隐变量。该方法引入两个新损失函数:注意力对比损失,通过确保每个自注意力区域仅对应一个主体的交叉注意力图来最小化不期望的重叠;注意力完整损失,最大化这些区域内的激活值以保证每个主体被完整且清晰地表示。我们的方法在噪声优化框架内运行,无需重新训练基础模型。在多个基准测试中,实验证明CoCoNO显著提升图文对齐性能,并超越当前最先进方法。

原文摘要 · Abstract (English)

Despite recent advancements in text-to-image models, achieving semantically accurate images in text-to-image diffusion models is a persistent challenge. While existing initial latent optimization methods have demonstrated impressive performance, we identify two key limitations: (a) attention neglect, where the synthesized image omits certain subjects from the input prompt because they do not have a designated segment in the self-attention map despite despite having a high-response cross-attention, and (b) attention interference, where the generated image has mixed-up properties of multiple subjects because of a conflicting overlap between cross- and self-attention maps of different subjects. To address these limitations, we introduce CoCoNO, a new algorithm that optimizes the initial latent by leveraging the complementary information within self-attention and cross-attention maps. Our method introduces two new loss functions: the attention contrast loss, which minimizes undesirable overlap by ensuring each self-attention segment is exclusively linked to a specific subject's cross attention map, and the attention complete loss, which maximizes the activation within these segments to guarantee that each subject is fully and distinctly represented. Our approach operates within a noise optimization framework, avoiding the need to retrain base models. Through extensive experiments on multiple benchmarks, we demonstrate that CoCoNO significantly improves text-image alignment and outperforms the current state of the art.

文本生成图像注意力机制扩散模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。