小模型中丢弃法仍能提升训练效率和拟合质量。
STLM Engineering Report: Dropout
- 在小数据下用丢弃法提高模型样本效率。
- 大数据下丢弃法仍能改善模型拟合，避免欠拟合。
- 发现现有解释不适用于语言建模中的丢弃机制。
本文研究了丢弃法（dropout）在现代语言模型中的有效性，特别关注参数量小于100M的模型。首先探讨其在小规模高质量数据集下的样本效率提升作用；其次分析其在大规模数据集上改善模型拟合、缓解欠拟合的能力。结果表明，与传统认知一致，丢弃法在过拟合场景下依然有效；且在数据充足时，仍可能提升模型性能，这与前期研究结论相符。然而，在语言建模任务中，现有对丢弃法增益机制的解释不再适用。
原文摘要 · Abstract (English)
In this work we explore the relevance of dropout for modern language models, particularly in the context of models on the scale of <100M parameters. We explore it's relevance firstly in the regime of improving the sample efficiency of models given small, high quality datasets, and secondly in the regime of improving the quality of its fit on larger datasets where models may underfit. We find that concordant with conventional wisdom, dropout remains effective in the overfitting scenario, and that furthermore it may have some relevance for improving the fit of models even in the case of excess data, as suggested by previous research. In the process we find that the existing explanation for the mechanism behind this performance gain is not applicable in the case of language modelling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。