arXiv:2411.07132cs.CVcs.AI2024-11NeurIPS被引 40

通过合并文本标记提升图像生成中的语义绑定能力,无需微调模型。

Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis

  • 将相关词元合并为复合词元,共享注意力图以增强语义关联。
  • 在多对象复杂提示下,绑定准确率显著优于现有方法。
  • 适合需要精准控制生成内容的视觉创作与工业应用。

尽管文本到图像(T2I)模型具备强大的生成能力,但常无法准确绑定输入提示中语义相关的物体或属性,这一问题称为语义绑定。以往方法要么需对整个T2I模型进行密集微调,要么要求用户或大语言模型指定生成布局,增加了复杂性。本文将语义绑定定义为属性绑定(物体与其属性关联)或对象绑定(物体与相关子物体关联)。提出一种新方法Token Merging(ToMe),通过聚合相关词元形成单一复合词元,确保物体、其属性及子物体共享相同的交叉注意力映射。为缓解复杂提示中主物体的混淆,引入末端词元替换作为补充策略。为进一步优化生成初期的布局阶段,加入熵损失和语义绑定损失,迭代更新复合词元以提升生成一致性。在T2I-CompBench及自建的GPT-4o对象绑定基准上进行了广泛实验,验证了ToMe在多物体、多属性复杂场景下的有效性,优于现有方法。代码将公开于 <https://github.com/hutaihang/ToMe>。

原文摘要 · Abstract (English)

Although text-to-image (T2I) models exhibit remarkable generation capabilities, they frequently fail to accurately bind semantically related objects or attributes in the input prompts; a challenge termed semantic binding. Previous approaches either involve intensive fine-tuning of the entire T2I model or require users or large language models to specify generation layouts, adding complexity. In this paper, we define semantic binding as the task of associating a given object with its attribute, termed attribute binding, or linking it to other related sub-objects, referred to as object binding. We introduce a novel method called Token Merging (ToMe), which enhances semantic binding by aggregating relevant tokens into a single composite token. This ensures that the object, its attributes and sub-objects all share the same cross-attention map. Additionally, to address potential confusion among main objects with complex textual prompts, we propose end token substitution as a complementary strategy. To further refine our approach in the initial stages of T2I generation, where layouts are determined, we incorporate two auxiliary losses, an entropy loss and a semantic binding loss, to iteratively update the composite token to improve the generation integrity. We conducted extensive experiments to validate the effectiveness of ToMe, comparing it against various existing methods on the T2I-CompBench and our proposed GPT-4o object binding benchmark. Our method is particularly effective in complex scenarios that involve multiple objects and attributes, which previous methods often fail to address. The code will be publicly available at \url{https://github.com/hutaihang/ToMe}.

文本生成语义绑定图像生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。