首次揭示文本生成图像模型中填充令牌的三种作用机制。
Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models
- 通过因果分析法研究填充令牌在不同环节的信息编码方式。
- 发现填充令牌可能影响文本编码、扩散过程或被模型忽略。
- 揭示其作用与注意力机制和训练策略密切相关，适合模型优化研究者阅读。
文本到图像（T2I）扩散模型依赖编码后的提示词引导图像生成。通常在文本编码前通过添加填充令牌将其扩展至固定长度。尽管这是默认做法，但填充令牌对生成过程的影响尚未被深入研究。本文首次系统分析了填充令牌在T2I模型中的作用。我们开发了两种因果分析技术，用于探究不同组件中令牌表示的信息编码方式。基于这些方法，我们考察了填充令牌在何时以及如何影响图像生成。研究发现存在三种不同场景：填充令牌可能在文本编码阶段、扩散过程中产生影响，或被模型有效忽略。此外，我们识别出这些现象与模型架构（交叉注意力或自注意力）及训练策略（冻结或可训练文本编码器）之间的关键关联。这些发现加深了对填充令牌工作机制的理解，可能为未来T2I系统的模型设计与训练实践提供指导。
原文摘要 · Abstract (English)
Text-to-image (T2I) diffusion models rely on encoded prompts to guide the image generation process. Typically, these prompts are extended to a fixed length by adding padding tokens before text encoding. Despite being a default practice, the influence of padding tokens on the image generation process has not been investigated. In this work, we conduct the first in-depth analysis of the role padding tokens play in T2I models. We develop two causal techniques to analyze how information is encoded in the representation of tokens across different components of the T2I pipeline. Using these techniques, we investigate when and how padding tokens impact the image generation process. Our findings reveal three distinct scenarios: padding tokens may affect the model's output during text encoding, during the diffusion process, or be effectively ignored. Moreover, we identify key relationships between these scenarios and the model's architecture (cross or self-attention) and its training process (frozen or trained text encoder). These insights contribute to a deeper understanding of the mechanisms of padding tokens, potentially informing future model design and training practices in T2I systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。