发现扩散模型中模板词元隐式充当语义寄存器,维持物体身份。
Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers

- 通过因果可解释性框架分离提示与模板词元,揭示其作用机制。
- 模板词元不直接读取提示,而是从图像潜变量中间接获取身份信息。
- 早期块确定身份、中间块传递、晚期块细化,适合研究模型内部机理者阅读。
现代文本到图像扩散变换器(DiTs)通过联合注意力生成图像,其中文本与图像词元在单一序列中直接交互。大规模DiTs的条件输入不仅包含用户提示,还包括由基于LLM的文本编码器引入的聊天模板词元。然而这些词元如何参与去噪计算仍不清楚。为此,我们提出一种因果可解释性框架,将提示内容词元与聊天模板词元分离。结果发现,模板词元在编码器输出中携带极少提示特定信息。但令人惊讶的是,它们成为图像到文本注意力的主要汇聚点,并在DiT中因果性地维持物体身份,扮演隐式语义寄存器角色。我们进一步发现,它们的身份是间接获得的:并非直接读取提示词元,而是从已注入提示语义的图像潜变量中获取。此外,我们揭示了注意力头与深度之间的分工:不同头负责语义路由或视觉结构生成,身份在早期块确定、中间块传递、晚期块细化。作为实际应用,该分析得出无需训练的剪枝规则,可移除无因果作用的提示读取头,减少20%的联合注意力计算量,仅损失1.4点GenEval准确率。整体上,本工作表明输入端的语义编码词元未必在生成过程中持续维护语义,同时为扩散变换器内部机制提供了因果视角。
原文摘要 · Abstract (English)
Modern text-to-image diffusion transformers (DiTs) generate images through joint attention, in which text and image tokens interact directly within a single sequence. In large-scale DiTs, the conditioning input contains not only the user prompt but also chat-template tokens introduced by LLM-based text encoders. Yet how these tokens participate in the denoising computation remains poorly understood. To probe this, we introduce a causal interpretability framework. Using it to separate prompt-content tokens from chat-template tokens, we find that the template tokens carry little prompt-specific information at the encoder output. Yet surprisingly, they emerge as dominant image-to-text attention sinks and causally maintain object identity inside the DiT, acting as implicit semantic registers. We show that they acquire this identity indirectly. Rather than reading the prompt tokens, they draw the identity from the image latents into which the prompt semantics have already been injected at the very first layer. We further reveal a division of labor across heads and depth in DiTs, where distinct heads route semantics or render visual structure, and identity is committed in early blocks, carried by middle blocks, and refined in late ones. As a practical payoff, this analysis yields a training-free pruning rule that removes the causally inert prompt-reading heads and cuts $20\%$ of joint-attention FLOPs at a $1.4$-point cost in GenEval accuracy. Overall, our work not only reveals that the tokens encoding semantics at the input need not be those that maintain them during generation, but also provides a causal view of internal mechanisms in diffusion transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。