提出FOCUS优化器，让大模型训练更稳更快。
FOCUS: First Order Concentrated Updating Scheme
- 用动量引导参数更新，增强对噪声的鲁棒性。
- 在GPT-2上比Signum更稳定，比Adam更快。
- 适合追求高效训练的大模型研究者。
大型语言模型（LLMs）表现卓越，改进其预训练过程是进一步提升能力的关键。基于Adam、学习率衰减和权重衰减的成功经验，我们假设预训练损失景观具有狭窄山谷结构。通过合成损失函数实验发现，当梯度查询噪声相对于山谷陡峭度较高时，Adam性能低于Signum，因其过度减小有效步长。这一观察促使我们提出FOCUS，一种在Signum基础上引入移动平均参数吸引力的优化器，使其在保持较大步长的同时更好地应对噪声。在GPT-2训练中，FOCUS表现出比Signum更稳定的性能，且比Adam更快。结果表明，梯度噪声可能是制约大模型训练的被低估因素，而FOCUS提供了有前景的解决方案。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate remarkable performance, and improving their pre-training process appears to be key to enhancing their capabilities further. Based on the documented success of Adam, learning rate decay, and weight decay, we hypothesize that the pre-training loss landscape features a narrowing valley structure. Through experiments with synthetic loss functions, we discover that when gradient query noise is high relative to the valley's sharpness, Adam's performance falls behind that of Signum because Adam reduces the effective step size too drastically. This observation led us to develop FOCUS, an optimizer that enhances Signum by incorporating attraction toward moving averaged parameters, allowing it to handle noise better while maintaining larger step sizes. In training GPT-2, FOCUS proves to be more stable than Signum and faster than Adam. These results suggest that gradient noise may be an underappreciated limiting factor in LLM training, and FOCUS offers promising solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。