将自回归模型转为扩散模型时，计算机制会随任务类型选择性保留或重构。
Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models
- 根据任务特性，扩散模型选择性保留或重排自回归计算路径。
- 全局约束任务中，关键计算提前至浅层，深度模式稳定。
- 适合研究模型机制演化与非自回归生成的科研人员。
将预训练自回归模型（ARMs）后训练为掩码扩散模型（MDMs）是实现扩散语言建模的有效途径，但其计算机制是否复用原有自回归结构仍不明确。我们在两个70亿参数的ARM-MDM系列上，通过四项受控诊断任务进行对比，发现机制转变具有任务依赖性。在前缀主导任务中，MDMs基本保留原有的高贡献路径，或仅发生轻微的计算位置变化；而在全局约束任务中，计算重组显著增强，任务相关计算向更浅层转移。该深度模式在提示重采样、电路预算和推理预算变化下均保持稳定，定向消融实验也证实了所识别结构的功能重要性。组件层面分析显示，ARMs更依赖高度特化的组件，而MDMs表现出较弱的单组件专一性及更强的输出空间对齐。结果表明，扩散后训练会根据任务结构选择性地保留或重构继承计算，而非统一替换自回归机制。
原文摘要 · Abstract (English)
Post-training pretrained autoregressive models (ARMs) into masked diffusion models (MDMs) provides an efficient route to diffusion language modeling, but it remains unclear whether the resulting models reuse inherited autoregressive computation or reorganize it for non-autoregressive generation. We compare two 7B ARM-MDM families across four controlled diagnostic tasks and find a task-dependent mechanism shift. On prefix-dominant tasks, MDMs largely preserve inherited high-attribution pathways or exhibit only modest changes in where computation occurs. On globally constrained tasks, the reorganization is substantially stronger, with task-relevant computation shifting toward earlier layers. This depth-wise pattern persists across prompt resampling, circuit budgets, and tested inference budgets, while targeted ablations support the functional importance of the identified structures under the tested intervention protocols. At the component level, diagnostic probes suggest that ARMs rely more strongly on sharply specialized components, whereas MDMs exhibit weaker single-component specialization and more diffuse output-space alignment. Together, these results suggest that diffusion post-training selectively preserves or reorganizes inherited computation according to task structure, rather than uniformly replacing autoregressive mechanisms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。