对比自回归与掩码扩散语言模型，发现生成方式影响训练效率和输出多样性。
Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison
- 在相同数据与算力下，对比自回归与掩码扩散两种生成范式
- 自回归模型更快收敛但更易重复，掩码扩散更慢但输出更丰富多样
- 适合关注生成多样性或可复现实验的研究者
我们对自回归（AR）与掩码扩散语言模型（MDLM）进行了受控的实证比较。两模型均在相同数据集（TinyStories，50M tokens）、相同计算预算（20,000步，批量32，序列长度512）和相同硬件（NVIDIA H100 80GB）下训练，仅生成范式为变量。结果显示：两者训练吞吐量相近（约5万词/秒），掩码扩散仅多耗时4.7%；自回归模型在第14,000步即开始过拟合，而掩码扩散在第20,000步仍持续优化，表明其最优训练策略不同；对1,000个生成样本的定量分析显示存在结构化多样性-流畅性权衡：自回归模型输出流畅但高度重复（99.8%以同一词开头），掩码扩散模型生成更多样叙事（93.4%的5词开头唯一，更高Distinct-n，更低Self-BLEU），但偶有语法错误。所有代码、训练检查点及数据管道均已开源。
原文摘要 · Abstract (English)
We present a controlled empirical comparison between autoregressive (AR) and masked diffusion (MDLM) language models. Both models are trained on identical data (50M tokens from TinyStories), identical compute budget (20,000 steps, batch size 32, sequence length 512), and identical hardware (NVIDIA H100 80GB), isolating the generation paradigm as the sole variable. We report three findings. First, both paradigms achieve comparable training throughput (~50K tokens/second), with MDLM requiring only 4.7% more wall-clock time. Second, AR converges faster and begins overfitting by step 14,000, while MDLM converges more slowly and is still improving at step 20,000, suggesting different compute-optimal training regimes. Third, quantitative diversity analysis over 1,000 generated samples reveals a structural diversity-fluency trade-off: AR produces fluent but repetitive outputs (99.8% begin with the same word), while MDLM generates more diverse narratives (93.4% unique 5-word openings, higher Distinct-n, lower Self-BLEU), at the cost of occasional grammatical inconsistencies. All code, trained checkpoints, and data pipelines are released for reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。