提出新训练策略，让掩码扩散语言模型训练速度提升4倍
Understanding and Accelerating the Training of Masked Diffusion Language Models
- 用钟形时间采样缓解语言局部性导致的训练慢问题
- 在LM1B数据集上训练速度提升约4倍，性能不变
- 适合想加速训练但不牺牲生成质量的研究者
掩码扩散模型（MDMs）作为语言建模的新范式，相比自回归模型（ARMs）训练更慢，限制其规模化应用。本文分析发现，语言的局部性特征——目标词的预测信息集中于邻近位置——是主因。为此提出钟形时间采样策略，有效缓解此问题。实验表明，在One Billion Word Benchmark（LM1B）上，采用该训练方案的MDM达到相同验证负对数似然（NLL）的速度快约4倍，且生成困惑度、零样本困惑度及下游任务表现均有更快提升。
原文摘要 · Abstract (English)
Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling. However, MDMs are known to learn substantially more slowly than ARMs, which may become problematic when scaling MDMs to larger models. Therefore, we ask the following question: how can we accelerate standard MDM training while maintaining its final performance? To this end, we first provide a detailed analysis of why MDM training is slow. We find that the main factor is the locality bias of language: the predictive information for a token is concentrated in nearby positions. We further investigate how this bias slows learning and suggest a simple yet effective remedy: bell-shaped time sampling as a training strategy. Notably, MDMs trained with our training recipe reach the same validation negative log-likelihood (NLL) up to $\sim4\times$ faster than standard training on One Billion Word Benchmark (LM1B). We also show faster improvements in generative perplexity, zero-shot perplexity, and downstream task performance on various benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。