通过分析文本嵌入几何特性，提升图文生成中语义绑定效果。
Geometrical Properties of Text Token Embeddings for Strong Semantic Binding in Text-to-Image Generation
- 利用文本词元嵌入的夹角和模长差异，增强跨名词短语注意力区分度。
- 无需训练，在多个数据集上显著改善复杂场景下的图文对齐精度。
- 适合关注图文生成语义准确性的研究者与开发者使用。
文本到图像（T2I）模型在包含多个对象和属性的复杂场景中常出现图文错位问题。虽然语义绑定方法尝试通过文本或潜在空间优化，结合交叉注意力（CA）映射来关联生成的属性与对应名词短语（NPs），但影响语义绑定的关键因素仍不明确。本文研究了文本词元嵌入及其CA映射的几何特性，发现词元嵌入的夹角距离与模长是区分CA映射的关键因素。基于此理论发现，提出无需训练的文本嵌入感知框架TokeBi，实现强语义绑定。TokeBi包含因果感知投影外（CAPO）以区分不同名词短语间的CA映射，以及自适应词元混合（ATM）以增强跨名词短语分离性，同时保持同一短语内的一致性。大量实验证明，TokeBi在多种基线和数据集上均优于现有方法。
原文摘要 · Abstract (English)
Text-to-image (T2I) models often suffer from text-image misalignment in complex scenes involving multiple objects and attributes. Semantic binding has attempted to associate the generated attributes and objects with their corresponding noun phrases (NPs) by text or latent optimizations with the modulation of cross-attention (CA) maps; yet, the factors that influence semantic binding remain underexplored. Here, we investigate the geometrical properties of text token embeddings and their CA maps. We found that the geometrical properties of token embeddings, specifically angular distances and norms, are crucial factors in the differentiation of the CA map. These theoretical findings led to our proposed training-free text-embedding-aware T2I framework, dubbed \textbf{TokeBi}, for strong semantic binding. TokeBi consists of Causality-Aware Projection-Out (CAPO) for distinguishing inter-NP CA maps and Adaptive Token Mixing (ATM) for enhancing inter-NP separation while maintaining intra-NP cohesion in CA maps. Extensive experiments confirm that TokeBi outperforms prior arts across diverse baselines and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。