探究掩码扩散语言模型的并行生成能力与顺序依赖问题
Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow
- 通过AFP和Kendall's tau量化模型并行度与生成顺序
- 8个主流模型在58项任务中表现落后于自回归模型
- 发现模型会根据任务自动调整生成顺序，适合复杂推理场景
掩码扩散语言模型（MDLMs）理论上可实现并行生成和任意顺序解码，但当前模型是否真正具备这些能力尚不明确。本文通过平均最终并行度（AFP）和Kendall's tau指标，从并行强度和生成顺序两个维度评估了8个主流MDLMs（最大达100B参数），覆盖知识、推理和编程共58个基准测试。结果表明，MDLMs仍落后于同等规模的自回归模型，主要因并行概率建模削弱了词元间依赖关系。同时，模型展现出适应性解码行为：其并行度与生成顺序随任务领域、推理阶段及输出正确性显著变化。在需‘反向信息’的任务（如数独）中，模型倾向于先填充较易空白，凸显其优势。最后，本文提出理论依据与设计思路，支持‘生成后编辑’范式，可在保留并行解码效率的同时缓解依赖损失。
原文摘要 · Abstract (English)
Masked Diffusion Language Models (MDLMs) promise parallel token generation and arbitrary-order decoding, yet it remains unclear to what extent current models truly realize these capabilities. We characterize MDLM behavior along two dimensions -- parallelism strength and generation order -- using Average Finalization Parallelism (AFP) and Kendall's tau. We evaluate eight mainstream MDLMs (up to 100B parameters) on 58 benchmarks spanning knowledge, reasoning, and programming. The results show that MDLMs still lag behind comparably sized autoregressive models, mainly because parallel probabilistic modeling weakens inter-token dependencies. Meanwhile, MDLMs exhibit adaptive decoding behavior: their parallelism and generation order vary significantly with the task domain, the stage of reasoning, and whether the output is correct. On tasks that require "backward information" (e.g., Sudoku), MDLMs adopt a solution order that tends to fill easier Sudoku blanks first, highlighting their advantages. Finally, we provide theoretical motivation and design insights supporting a Generate-then-Edit paradigm, which mitigates dependency loss while retaining the efficiency of parallel decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。