nGPT通过球面约束提升训练效率，14B模型用一半数据达相同效果。
Training nGPT
- 参数与激活向量约束在单位超球面，实现球面表示学习。
- 140亿参数模型仅需原模型一半训练数据即达相同验证损失。
- 适用于大规模混合架构，尤其适合追求训练效率的研究者。
归一化Transformer（nGPT）通过将模型参数向量和激活向量约束在单位超球面上，实现超球面表示学习。本文提出一种nGPT的实用训练方案，并在包含140亿总参数的现代混合Mamba-2--Transformer混合专家（MoE）模型上进行评估。该方案引入了对数梯度预处理、对数学习率衰减、GatedAdamW、角度更新控制以及可选探索机制。与使用AdamW训练的同构混合MoE模型相比，140亿参数的nGPT模型在约一半训练标记下即可达到相同的验证损失。该训练方案在所考察的模型中均具可扩展性，最高支持140亿总参数。
原文摘要 · Abstract (English)
The normalized Transformer (nGPT) realizes hyperspherical representation learning by constraining model parameter vectors and activation vectors to the unit hypersphere. In this paper, we describe a practical training recipe for nGPT and evaluate it on modern hybrid Mamba-2--Transformer Mixture-of-Experts (MoE) models. The recipe introduces Logit Gradient Preconditioning, Logarithmic Learning Rate Decay, GatedAdamW, angular update control, and optional exploration mechanisms. Compared with an unnormalized model of the same hybrid MoE architecture trained with AdamW, the 14B-total-parameter nGPT model reaches the same validation loss using approximately half as many training tokens. The recipe scales across the models considered, which contain up to 14B total parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。