发现离散扩散语言模型的训练效率依赖噪声类型，均匀噪声更省数据。
Scaling Behavior of Discrete Diffusion Language Models
- 通过调节噪声类型研究扩散模型的缩放规律，关注批量和学习率影响。
- 均匀扩散在计算效率上优于掩码扩散，需更少数据、更多参数。
- 训练了100亿参数模型，达到10²²浮点运算，是目前最大公开均匀扩散模型。
现代大语言模型预训练消耗巨大算力与数据，不同模型的缩放行为成为关键区分因素。离散扩散语言模型（DLMs）作为自回归语言模型（ALMs）的替代方案被提出，但其缩放行为尚未充分探索，先前研究认为其需更多数据和算力才能达到ALMs性能。本文通过平滑插值掩码与均匀扩散噪声，细致研究不同噪声类型下的缩放规律，重点关注批量大小与学习率等超参数。实验表明，DLM的缩放行为强烈依赖于噪声类型，显著区别于ALMs。所有噪声类型在算力受限情况下均收敛至相似损失值，但均匀扩散在算力效率上更优，需更少数据、更多参数，适合数据受限场景。我们将均匀扩散模型扩展至100亿参数，训练达10²² FLOPs，验证了预测的缩放规律，成为迄今最大的公开均匀扩散模型。
原文摘要 · Abstract (English)
Modern LLM pre-training consumes vast amounts of compute and training data, making the scaling behavior, or scaling laws, of different models a key distinguishing factor. Discrete diffusion language models (DLMs) have been proposed as an alternative to autoregressive language models (ALMs). However, their scaling behavior has not yet been fully explored, with prior work suggesting that they require more data and compute to match the performance of ALMs. We study the scaling behavior of DLMs on different noise types by smoothly interpolating between masked and uniform diffusion while paying close attention to crucial hyperparameters such as batch size and learning rate. Our experiments reveal that the scaling behavior of DLMs strongly depends on the noise type and is considerably different from ALMs. While all noise types converge to similar loss values in compute-bound scaling, we find that uniform diffusion requires more parameters and less data for compute-efficient training compared to masked diffusion, making them a promising candidate in data-bound settings. We scale our uniform diffusion model up to 10B parameters trained for $10^{22}$ FLOPs, confirming the predicted scaling behavior and making it the largest publicly known uniform diffusion model to date.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。