研究语音转文本模型学习率预热策略，发现高温预热加速初期收敛但不影响最终性能。
The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence
- 提出子指数型学习率预热，适配大规模语音转文本训练
- 高温预热可加快初始收敛，但不提升最终模型精度
- 验证不同预热方案对性能影响，揭示预热阶段的局限性
大规模模型训练不仅资源消耗大，还面临收敛难题。在语音转文本（S2T）任务中，由于使用Conformer或Branchformer等复杂Transformer变体，单纯降低学习率已不足以解决问题。为此，OWSM采用双线性预热策略：先将学习率升至极小值，再提高到较高水平。该方法虽有效，但未与其它方案对比，也未系统研究不同预热策略对最终性能的影响。本文填补这一空白，发现：(i) 大规模S2T训练需要子指数型学习率预热；(ii) 预热阶段采用更高学习率虽能加速初期收敛，但不会提升最终模型性能。
原文摘要 · Abstract (English)
Training large-scale models presents challenges not only in terms of resource requirements but also in terms of their convergence. For this reason, the learning rate (LR) is often decreased when the size of a model is increased. Such a simple solution is not enough in the case of speech-to-text (S2T) trainings, where evolved and more complex variants of the Transformer architecture -- e.g., Conformer or Branchformer -- are used in light of their better performance. As a workaround, OWSM designed a double linear warmup of the LR, increasing it to a very small value in the first phase before updating it to a higher value in the second phase. While this solution worked well in practice, it was not compared with alternative solutions, nor was the impact on the final performance of different LR warmup schedules studied. This paper fills this gap, revealing that i) large-scale S2T trainings demand a sub-exponential LR warmup, and ii) a higher LR in the warmup phase accelerates initial convergence, but it does not boost final performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。