动态调整丢弃率，让Transformer训练更快更高效
Enhancing Transformer Training Efficiency with Dynamic Dropout
- 根据训练轮次或验证损失动态变化丢弃率
- 在Shakespeare_char数据集上加速训练并提升推理效率
- 适合追求训练速度与模型性能平衡的研究者
我们提出动态丢弃（Dynamic Dropout），一种新型正则化技术，通过根据训练轮次或验证损失改善情况动态调整Transformer模型的丢弃率，以提升训练效率。该方法解决了正则化强度与模型容量之间的权衡难题，对实现快速收敛和高性能至关重要。我们修改GPT模型以支持可变丢弃率，并在训练中使用线性衰减、指数衰减及基于验证损失的调整策略更新丢弃层。在Shakespeare_char数据集上的大量实验表明，相较于固定丢弃率的基线模型，动态丢弃显著加速了训练过程并提升了推理效率。基于验证损失的调整策略表现最佳，凸显了该技术在大规模Transformer模型训练中的潜力。
原文摘要 · Abstract (English)
We introduce Dynamic Dropout, a novel regularization technique designed to enhance the training efficiency of Transformer models by dynamically adjusting the dropout rate based on training epochs or validation loss improvements. This approach addresses the challenge of balancing regularization and model capacity, which is crucial for achieving fast convergence and high performance. Our method involves modifying the GPT model to accept a variable dropout rate and updating dropout layers during training using schedules such as linear decay, exponential decay, and validation loss-based adjustments. Extensive experiments on the Shakespeare\_char dataset demonstrate that Dynamic Dropout significantly accelerates training and improves inference efficiency compared to a baseline model with a fixed dropout rate. The validation loss-based adjustment schedule provided the best overall performance, highlighting the potential of Dynamic Dropout as a valuable technique for training large-scale Transformer models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。