扩散语言模型为何高效利用数据？关键在随机遮蔽与正则化。
What Makes Diffusion Language Models Super Data Learners?
- 通过随机遮蔽输入词元提升数据效率
- MLP dropout与权重衰减也能带来类似增益
- 适合关注训练效率与正则化机制的研究者
近期研究表明，扩散语言模型在数据有限条件下表现出卓越的数据效率，但其内在机制仍不明确。本文通过大量消融实验，分离出效率来源。结果表明，输入词元的随机遮蔽起主导作用。进一步发现，类似效果可通过 MLP dropout 和权重衰减实现，表明随机正则化在多轮训练中普遍提升数据效率。代码已公开于 https://github.com/zitian-gao/data-efficiency。
原文摘要 · Abstract (English)
Recent studies have shown that diffusion language models achieve remarkable data efficiency under limited-data constraints, yet the underlying mechanisms remain unclear. In this work, we perform extensive ablation experiments to disentangle the sources of this efficiency. Our results show that random masking of input tokens plays the dominant role. We further show that similar gains can be obtained through in MLP dropout and weight decay, indicating that stochastic regularization broadly enhances data efficiency in multi-epoch training. Our code is available at https://github.com/zitian-gao/data-efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。