扩散语言模型在数据稀缺时比自回归模型更擅长学习。
Diffusion Language Models are Super Data Learners
- 采用迭代双向去噪实现任意顺序建模与密集计算
- 1.7B参数模型在100亿唯一代码上超越对标自回归模型
- 小模型仅用10亿训练数据即达56%的HellaSwag准确率
在严格控制的预训练条件下，我们观察到一个拐点：当唯一数据有限时，扩散语言模型（DLMs）通过更多轮次训练持续优于自回归（AR）模型。该拐点随数据量或质量增加而延后，随模型规模增大而提前，并在稠密与稀疏架构间保持稳定。收益源于三个叠加因素：(1) 任意顺序建模，(2) 迭代双向去噪带来的超密集计算，(3) 内置蒙特卡洛增强；输入或参数噪声虽能提升AR模型在数据受限下的表现，但无法弥合差距。在规模化下，一个1.7B参数的DLM在100亿唯一Python token上使用约1.5万亿词元计算预算，超越了设定完全匹配的AR编码器。此外，一个1B参数的DLM仅用10亿训练数据，无需特殊技巧，在HellaSwag上达到超过56%准确率，在MMLU上超过33%。
原文摘要 · Abstract (English)
Under strictly controlled pre-training settings, we observe a Crossover: when unique data is limited, diffusion language models (DLMs) consistently surpass autoregressive (AR) models by training for more epochs. The crossover shifts later with more or higher-quality data, earlier with larger models, and persists across dense and sparse architectures. We attribute the gains to three compounding factors: (1) any-order modeling, (2) super-dense compute from iterative bidirectional denoising, and (3) built-in Monte Carlo augmentation; input or parameter noise improves AR under data constraint but cannot close the gap. At scale, a 1.7B DLM trained with a ~1.5T-token compute budget on 10B unique Python tokens overtakes an AR coder trained with strictly matched settings. In addition, a 1B-parameter DLM achieves > 56% accuracy on HellaSwag and > 33% on MMLU using only 1B tokens, without any special tricks, just by repeating standard pre-training data. We also show that rising validation cross-entropy does not imply degraded downstream performance in this regime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。