文本到图像模型其实只需要词义和词序,不依赖复杂语义上下文。
Text-to-Image Models Need Less from Text Encoders Than You Think

- 用仅含词义和位置的简化嵌入替代完整文本编码
- 生成图像质量与原文本编码效果相当
- 适合想压缩模型或理解生成机制的研究者
文本到图像模型通过文本提示作为人类意图的主要接口,提示由文本编码器转化为嵌入以指导图像生成。除了单个词的含义,文本嵌入还包含跨完整提示的上下文信息,如组合性和属性绑定。然而,图像模型是否真正利用这些更丰富的信息仍不清楚。本文探究:哪些文本表征特征对图像生成至关重要?我们发现,基于扩散的文本到图像变换器模型通常仅依赖文本表征的两个简单方面:(i) 相邻词元合并为单词表示(用于多词元词),(ii) 由文本编码器位置嵌入所保留的词序。为此,我们构建了一种新嵌入:仅编码单个词义和词序,但缺乏完整提示的上下文信息。结果表明,这种仅含位置标记词袋的表示足以成功引导图像生成,在视觉质量和文本一致性上与使用完整嵌入的效果相当。这说明,与普遍认知相反,文本到图像模型通常并不使用文本嵌入中超出词义和词序的丰富信息;复杂语言结构的解码实际上由图像模型自身完成。
原文摘要 · Abstract (English)
Text-to-image models rely on text prompts as their primary interface to human intent. Prompts are encoded by a text encoder into embeddings that condition the image generation process. Beyond individual token meanings, text embeddings encode contextual information across the full prompt, such as compositionality and attribute binding. However, whether image models actually exploit this richer information remains underexplored. Here, we address the question: Which aspects of text representation are essential for image generation? We show that text-to-image diffusion transformer-based models commonly rely only on two relatively straightforward aspects of text representations: (i) the merging of adjacent tokens into a word representation, for words spanning multiple tokens, and (ii) word order, which is imprinted by the positional embedding of the text-encoder. To show this, we construct a new text embedding that encodes only individual word meanings and order but lacks any contextual information about the full prompt. We find that this bag of position-tagged words representation is sufficient to successfully guide image generation, achieving visual quality and text fidelity that are on par with full text embedding-guided generation. This demonstrates that, contrary to common belief, text-to-image models often do not use the rich information encoded in the text embedding beyond individual word meanings and word order. Instead, the decoding of complex linguistic structures is performed by the image model itself. Project webpage: https://nsping13.github.io/contextless-TTI/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。