arXiv:2607.15893cs.CLcs.AI2026-07

发现扩散语言模型具备双向归纳能力,可同时利用前后文匹配答案。

Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models

论文配图:Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models
图 1 · 摘自论文原文
  • 通过注意力机制构建双向上下文记忆回路
  • 在双侧上下文可见时归纳性能超越自回归模型
  • 无需显式时间嵌入即可隐式计算掩码比例

尽管自回归(AR)Transformer 的内部机制已得到广泛研究,但新兴的扩散语言模型(DLMs)——通过迭代去噪生成文本——的机理仍不清晰。本文对比了注意力仅限于过去(AR)与吸收掩码(absorbing-mask)DLMs,在结构相同的条件下研究其归纳能力。结果表明,DLMs 学习到一种双向归纳回路:前向和后向注意力头将局部上下文写入残差流,后续归纳头则据此定位并复制匹配位置的答案。该回路具有方向对称性,无论源序列在前或在后均有效。当仅能看见左侧上下文时,其归纳表现不及对应自回归模型;但当掩码词两侧上下文皆可见时,性能显著更强,表明其依赖双向上下文而非单边增强。此外,我们提供了因果证据,显示 DLMs 能隐式计算全局掩码比例,并将其用作隐式时间步,尽管未提供显式时间嵌入。

原文摘要 · Abstract (English)

While the internal mechanisms of autoregressive (AR) transformers have been studied extensively, much less is known about diffusion language models (DLMs), an emerging alternative that generates text by iterative denoising. In this work, we study how DLMs implement induction, a mechanism behind in-context learning in which the model finds a repeated context and copies the token that followed it. Our analysis compares attention-only AR models and absorbing-mask DLMs with matched architectures. We find that DLMs learn a bidirectional induction circuit, where previous-token and next-token heads write local context into the residual stream and later induction heads use it to find and copy the answer from the matching source position. The circuit is direction-symmetric, working whether the source appears in the past or in the future. When only left context is visible, matching what an AR model sees, the DLM does not outperform its AR counterpart in induction capabilities. However, we observe it has stronger induction when both sides of the masked token are visible, pointing to bidirectional context access rather than a stronger one-sided mechanism. Beyond induction, we provide causal evidence that DLMs compute the global fraction of masked tokens and use it as an implicit timestep, even though they are given no explicit timestep embedding.

扩散模型上下文学习双向归纳

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。