探究丢弃率与残差连接的协同效应，提升语言模型训练效果
Investigating the Synergistic Effects of Dropout and Residual Connections on Language Model Training
- 调整各层与残差连接的丢弃率，研究其对训练的影响
- 发现残差深度与丢弃率存在影响收敛和泛化的权衡
- 适合关注模型训练稳定性和泛化能力的研究者
本文研究了丢弃技术在语言模型训练中缓解过拟合的关键作用。通过在经典Tiny Shakespeare数据集上训练解码器实现，系统考察了不同丢弃率对各层及残差连接的影响。实验表明，丢弃有助于正则化，残差连接促进收敛，二者存在显著协同效应。研究揭示了残差连接深度与连接处丢弃率之间的关键权衡关系，对实现深层神经网络的最优收敛与泛化具有重要意义。
原文摘要 · Abstract (English)
This paper examines the pivotal role of dropout techniques in mitigating overfitting in language model training. It conducts a comprehensive investigation into the influence of variable dropout rates on both individual layers and residual connections within the context of language modeling. Our study conducts training of a decoder implementation on the classic Tiny Shakespeare data to examine the effects of the adjustments on training efficiency and validation error. Results not only confirm the benefits of dropout for regularization and residuals for convergence, but also reveal their interesting interactions. There exists an important trade-off between the depth of residual connections and the dropout on these connections for optimal deep neural network convergence and generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。