nGPT通过球面归一化加速模型训练，提升学习效率。
nGPT: Normalized Transformer with Representation Learning on the Hypersphere
- 所有向量在单位球面上归一化，信息在球面上传播。
- 训练步数减少4到20倍，序列越长优势越明显。
- 适合追求高效训练的NLP与序列建模研究者。
我们提出一种新型神经网络架构——球面表示学习的归一化Transformer（nGPT）。在nGPT中，嵌入、MLP、注意力矩阵和隐藏状态的所有向量均进行单位范数归一化。输入标记流在高维球面表面传播，每一层通过MLP和注意力模块产生指向目标输出的位移。这些位移由位于同一球面的向量分量定义。实验表明，nGPT学习速度显著加快，在相同准确率下所需训练步数减少4至20倍，具体取决于序列长度。
原文摘要 · Abstract (English)
We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere. In nGPT, all vectors forming the embeddings, MLP, attention matrices and hidden states are unit norm normalized. The input stream of tokens travels on the surface of a hypersphere, with each layer contributing a displacement towards the target output predictions. These displacements are defined by the MLP and attention blocks, whose vector components also reside on the same hypersphere. Experiments show that nGPT learns much faster, reducing the number of training steps required to achieve the same accuracy by a factor of 4 to 20, depending on the sequence length.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。