分析文本生成图像中词语间信息流动,发现关键词编码集中且易误关联。
Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models
- 通过打孔法分析文本编码,发现语义信息多集中于少数关键词。
- 如'Golden Gate Bridge'中仅'Gate'即能代表整词,其余可忽略。
- 词语间常孤立编码,但局部上下文会引发错误关联,如'pool'被误为'台球桌'
文本到图像生成模型存在对齐问题,即生成图像无法准确反映文本提示中的物体与关系。现有工作聚焦优化扩散过程,忽视了引导扩散的文本编码器的作用。本文从两个层面研究文本提示中语义信息在词元表示间的分布:(1) 词内表示——单个词元是否表征其词汇项(即表达单一概念的词或短语);(2) 词间交互——不同词汇项的词元之间是否存在信息流动。采用打孔技术揭示编码模式,发现信息通常集中在一两个词元中;例如,在‘San Francisco's Golden Gate Bridge’中,‘Gate’足以捕捉整个表达,其余词元可被有效丢弃。词汇项也倾向于保持孤立;如在‘a green dog’中,‘dog’不包含‘green’的视觉信息。然而,在某些情况下,词元间存在相互影响,常导致误解——例如在‘a pool by a table’中,‘pool’经上下文整合后被编码为‘pool table’。研究结果凸显词元级编码在图像生成中的关键作用,并证明在编码阶段进行简单干预即可显著提升对齐性与生成质量。
原文摘要 · Abstract (English)
Text-to-image generation models suffer from alignment problems, where generated images fail to accurately capture the objects and relations in the text prompt. Prior work has focused on improving alignment by refining the diffusion process, ignoring the role of the text encoder, which guides the diffusion. In this work, we investigate how semantic information is distributed across token representations in text-to-image prompts, analyzing it at two levels: (1) in-item representation-whether individual tokens represent their lexical item (i.e., a word or expression conveying a single concept), and (2) cross-item interaction-whether information flows between tokens of different lexical items. We use patching techniques to uncover encoding patterns, and find that information is usually concentrated in only one or two of the item's tokens; for example, in the item ``San Francisco's Golden Gate Bridge'', the token ``Gate'' sufficiently captures the entire expression while the other tokens could effectively be discarded. Lexical items also tend to remain isolated; for instance, in the prompt ``a green dog'', the token ``dog'' encodes no visual information about ``green''. However, in some cases, items do influence each other's representation, often leading to misinterpretations-e.g., in the prompt ``a pool by a table'', the token ``pool'' represents a ``pool table'' after contextualization. Our findings highlight the critical role of token-level encoding in image generation, and demonstrate that simple interventions at the encoding stage can substantially improve alignment and generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。