DiffusionGemma的生成顺序既非并行也非自回归,而是依赖分析粒度的局部左向偏置。
Neither Parallel Nor Sequential: How DiffusionGemma Actually Commits Tokens

- 通过记录采样器提交位置和置信度,揭示解码顺序受分析粒度影响
- 在686个提示中,模型以大批次同时提交token,内部顺序本质未定义
- 适合关注生成机制真实性、模型行为解释的研究者
开放扩散语言模型常被宣传为并行、非自回归解码器,但其实际提交token的顺序极少被测量。我们对基于Gemma 4的260亿参数掩码离散扩散专家混合模型DiffusionGemma进行插桩,记录采样器接受步骤中的提交位置、时间与置信度。在包含686个提示和六种解码场景的测试中发现,其解码既非完全并行,也非分块自回归:存在部分从左到右的提交偏置,其强度几乎完全取决于分析粒度。逐个token看时顺序很弱,随粒度粗化而平滑增强,因此模型的“块大小”实为测量尺度的产物而非架构特性。模型以大批次同时提交,批次内顺序本质上未定义而非仅未观测。行为具有场景依赖性:结构化JSON以近乎任意顺序提交;位置提交置信度与数学推理正确性相关,但对事实回忆无预测力。提交行为激进,在步数预算内快速完成,且任务准确率与自回归Gemma-4相当。此外,本研究核心贡献是方法论:诚实测量解码顺序需处理尾部EOS填充、组内混淆、提交非单调性、块大小敏感性及大规模提交批次并列等问题,否则可能制造出虚假的解码顺序结论。
原文摘要 · Abstract (English)
Open diffusion language models are marketed as parallel, non-autoregressive decoders, yet the order in which a shipped checkpoint actually commits its tokens is almost never measured. We instrument DiffusionGemma 26B, a masked discrete-diffusion mixture-of-experts model built on Gemma 4, hooking its sampler's accept step to record which canvas positions commit, when, and at what confidence. Across a 686-prompt, six-regime probe suite we find that its decoding is neither parallel nor block-autoregressive: it follows a partial left-to-right commit bias whose apparent strength depends almost entirely on the granularity at which you look. Order is weak token by token and strengthens smoothly as the analysis is coarsened, so the model's "block size" turns out to be an artifact of the measuring ruler rather than the architecture. The model commits in large simultaneous batches, leaving much of the within-batch order genuinely undefined rather than merely unobserved. The behaviour is regime-dependent: structured JSON is committed in essentially arbitrary order, and a position's commit confidence tracks correctness on mathematical reasoning but carries no signal on factual recall. Commitment is aggressive, finishing in a short late burst well inside the step budget, while task accuracy matches the model's autoregressive Gemma-4 sibling. Beyond these findings, our central contribution is methodological: measuring decoding order honestly demands handling trailing-EOS padding, within-regime confounding, commit non-monotonicity, block-size sensitivity, and large commit-batch ties, each of which can otherwise manufacture a decoding-order result that is not really there.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。