提出两种新模型，让扩散语言模型更稳定地完成有顺序的任务。
On the Trainability of Masked Diffusion Language Models via Blockwise Locality
- 设计块内自回归结构，注入左右依赖性提升训练稳定性
- 在线性回归任务上达到与自回归模型相当的训练稳定度
- 适合需要规划能力又要求稳定训练的结构化生成场景
掩码扩散语言模型（MDMs）作为自回归大语言模型（AR-LLMs）的潜在替代方案，其优化常不稳定。本文在三个受控任务中对比块状MDMs与AR-LLMs：上下文线性回归、图路径查找和数独求解。发现标准随机掩码MDMs无法可靠学习线性回归，路径查找时训练方差高，但在数独上优于AR-LLMs。为缓解不稳定性，提出两种感知局部性的块状模型——Jigsaw与Scatter，通过在块内强制自回归局部性，同时保留块级迭代优化。实验表明，Jigsaw在回归任务上达到与AR-LLM相当的稳定性，且保持数独优势；Scatter则维持扩散模型在路径查找中的规划能力。结果表明，即使采用块状结构，随机掩码仍非有序生成的理想扩散语言模型实现方式，亟需超越随机掩码的新模型设计。
原文摘要 · Abstract (English)
Masked diffusion language models (MDMs) have recently emerged as a promising alternative to standard autoregressive large language models (AR-LLMs), yet their optimization can be substantially less stable. We study blockwise MDMs and compare them with AR-LLMs on three controlled tasks that stress different aspects of structured generation: in-context linear regression, graph path-finding, and Sudoku solving. We find that standard random-masking MDMs fail to reliably learn linear regression, exhibit high variance training dynamics on graph path-finding, while outperforming AR-LLMs on Sudoku. To mitigate these instabilities, we propose two locality aware blockwise models, namely Jigsaw and Scatter, that inject left-to-right inductive bias by enforcing autoregressive locality within blocks while preserving iterative refinement at the block level. Empirically, Jigsaw matches AR-LLM stability on linear regression and remains strong on Sudoku, while Scatter retains diffusion's planning advantage on path-finding. Our results indicate that standard random-masking MDMs, even with blockwise variants, may be a suboptimal instantiation of diffusion LMs for ordered generation, motivating models beyond random masking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。