去掉丢弃法反而提升单轮预训练语言模型性能。
Drop Dropout on Single-Epoch Language Model Pretraining
- 在单轮预训练中不使用丢弃法，能提升模型表现。
- 不使用丢弃法时，下游任务如问答、自然语言推理表现更好。
- 适合追求高效预训练的模型开发者，尤其关注可编辑性场景。
最初，丢弃法被视作减少过拟合、提升深度学习性能的关键正则化技术。然而，现代大语言模型常用的单轮预训练任务本身过拟合极少，导致丢弃法未被广泛采用。尽管如此，其在语言模型预训练中的作用尚无充分实证研究。我们通过在掩码（BERT）和自回归（Pythia 160M 和 1.4B）语言模型上进行单轮预训练实验，测试不同丢弃率的影响，发现不使用丢弃法时，语言建模、形态句法（BLiMP）、问答（SQuAD）和自然语言推断（MNLI）等下游任务性能均有所提升。此外，近期提出的“早丢弃”策略也表现不如完全不使用丢弃法。我们还评估了模型可编辑性，发现无丢弃训练的模型在基于梯度的编辑（MEND）中更成功，而在基于表示的编辑（ReFT）中表现相当。因此，我们建议在单轮预训练中去除丢弃法。
原文摘要 · Abstract (English)
Originally, dropout was seen as a breakthrough regularization technique that reduced overfitting and improved performance in almost all applications of deep learning by reducing overfitting. Yet, single-epoch pretraining tasks common to modern LLMs yield minimal overfitting, leading to dropout not being used for large LLMs. Nevertheless, no thorough empirical investigation has been done on the role of dropout in LM pretraining. Through experiments in single-epoch pretraining of both masked (BERT) and autoregressive (Pythia 160M and 1.4B) LMs with varying levels of dropout, we find that downstream performance in language modeling, morpho-syntax (BLiMP), question answering (SQuAD), and natural-language inference (MNLI) improves when dropout is not applied during pretraining. We additionally find that the recently-introduced "early dropout" also degrades performance over applying no dropout at all. We further investigate the models' editability, and find that models trained without dropout are more successful in gradient-based model editing (MEND) and equivalent in representation-based model editing (ReFT). Therefore, we advocate to drop dropout during single-epoch pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。