改进Transformer学习率迁移，实现模型规模与序列长度间高效泛化。
Learning Rate Transfer in Normalized Transformers
- 基于对齐指数重构超参数传递机制，提出新型νGPT参数化方法。
- 在不同宽度、深度和序列长度下均实现学习率迁移，训练效率显著提升。
- 适合大规模语言模型训练，尤其关注超参数可迁移性的研究者。
归一化Transformer（nGPT）在arXiv:2410.01131中实现了显著的训练加速，且无需权重衰减或学习率预热。然而，尽管其超参数显式随模型规模缩放，我们观察到nGPT在模型维度和词元跨度上并未表现出学习率迁移。为解决此问题，本文结合数值实验与对齐指数（arXiv:2407.05872）的严谨应用，重新审视并改进了μP超参数传递方法（arXiv:2011.14522）。由此提出新型nGPT参数化形式νGPT。通过广泛的实证验证，发现νGPT在模型宽度、深度及词元跨度之间均能实现学习率迁移。
原文摘要 · Abstract (English)
The Normalized Transformer, or nGPT (arXiv:2410.01131) achieves impressive training speedups and does not require weight decay or learning rate warmup. However, despite having hyperparameters that explicitly scale with model size, we observe that nGPT does not exhibit learning rate transfer across model dimension and token horizon. To rectify this, we combine numerical experiments with a principled use of alignment exponents (arXiv:2407.05872) to revisit and modify the $μ$P approach to hyperparameter transfer (arXiv:2011.14522). The result is a novel nGPT parameterization we call $ν$GPT. Through extensive empirical validation, we find $ν$GPT exhibits learning rate transfer across width, depth, and token horizon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。