通过平均训练过程中的模型权重，可加速训练并小幅提升泛化性能。
When, Where and Why to Average Weights?
- 在训练轨迹上平均权重，无需复杂修改即可提升效率
- 实验显示训练速度显著加快，泛化性能略有提升
- 适合追求高效训练的深度学习实践者
沿训练轨迹平均检查点是一种简单而有效的提升机器学习模型泛化能力并缩短训练时间的方法。为全面评估该技术的潜力，我们基于AlgoPerf基准框架，在七种架构和数据集上进行了大规模实验。结果表明，权重平均能显著加速训练，带来可观的效率提升，仅需极小的实现与内存开销，且对所有任务均能轻微改善泛化性能。此外，我们分析了权重平均与学习率衰减的关系，提出最优结合策略以获得最佳表现。
原文摘要 · Abstract (English)
Averaging checkpoints along the training trajectory is a simple yet powerful approach to improve the generalization performance of Machine Learning models and reduce training time. Motivated by these potential gains, and in an effort to fairly and thoroughly benchmark this technique, we present an extensive evaluation of averaging techniques in modern Deep Learning, which we perform using AlgoPerf \citep{dahl_benchmarking_2023}, a large-scale benchmark for optimization algorithms. We investigate whether weight averaging can reduce training time, improve generalization, and replace learning rate decay, as suggested by recent literature. Our evaluation across seven architectures and datasets reveals that averaging significantly accelerates training and yields considerable efficiency gains, at the price of a minimal implementation and memory cost, while mildly improving generalization across all considered workloads. Finally, we explore the relationship between averaging and learning rate annealing and show how to optimally combine the two to achieve the best performances.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。