提出CompleteP参数化，让大模型训练更高效省算力
Don't be lazy: CompleteP enables compute-efficient deep transformers
- 设计新参数化方法，实现深度变化时超参自动迁移
- 在所有层中避免懒惰学习，充分使用模型深度与非线性
- 比现有最佳方案提升12-34%算力效率，适合不同硬件
我们研究大模型训练中的算力效率问题，关注模型规模变化时参数化策略对模型和优化器超参数（HPs）调整的影响。部分参数化无法在模型深度变化时转移最优基础超参（如学习率），导致需重新调参（代价高），或接受次优训练效果。即使能转移超参，理论分析表明仍可能处于懒惰学习区，各层仅学习接近线性化的特征，限制了深度与非线性的有效利用。本文提出并采用称为CompleteP的参数化方法，同时实现跨深度的超参可迁移性和所有层的非懒惰学习。CompleteP使更广泛的宽度/深度比保持算力高效，适配不同硬件与运行场景，并在所有实验（基于Cerebras CS-3系统）中实现12-34%的算力效率提升。最小实现已开源：https://github.com/EleutherAI/nanoGPT-mup/tree/completep。
原文摘要 · Abstract (English)
We study compute efficiency of LLM training when using different parameterizations, i.e., rules for adjusting model and optimizer hyperparameters (HPs) as model size changes. Some parameterizations fail to transfer optimal base HPs (such as learning rate) across changes in model depth, requiring practitioners to either re-tune these HPs as they scale up (expensive), or accept sub-optimal training when re-tuning is prohibitive. Even when they achieve HP transfer, we develop theory to show parameterizations may still exist in the lazy learning regime where layers learn only features close to their linearization, preventing effective use of depth and nonlinearity. Finally, we identify and adopt the parameterization we call CompleteP that achieves both depth-wise HP transfer and non-lazy learning in all layers. CompleteP enables a wider range of model width/depth ratios to remain compute-efficient, unlocking shapes better suited for different hardware settings and operational contexts. Moreover, CompleteP enables 12-34% compute efficiency improvements over the prior state-of-the-art. All experiments were run on Cerebras CS-3 systems. A minimal implementation is available at https://github.com/EleutherAI/nanoGPT-mup/tree/completep.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。