nGPT架构天然支持4比特训练，无需额外校正。
Normalized Architectures are Natively 4-Bit
- 通过约束权重和隐藏层在单位超球面，提升低精度计算鲁棒性。
- 1.2B、3B及30B模型均实现稳定端到端NVFP4训练，性能无损。
- 适合追求高效推理的LLM开发者，尤其大模型场景。
在4比特精度下训练大语言模型对效率至关重要。我们发现，nGPT是一种将权重和隐藏表示限制在单位超球面的架构，天生对低精度算术更具鲁棒性。这无需引入随机Hadamard变换或张量级缩放等干预措施即可保持模型质量，并支持稳定的端到端NVFP4训练。我们在1.2B稠密模型以及最大达3B/30B参数的混合（Mamba-Transformer）MoE模型上验证了该方法。我们追溯其鲁棒性源于点积：尽管标准与归一化架构中量化噪声均呈弱相关，但信号行为不同。在nGPT中，超球面约束增强了元素乘积间的微弱正相关，使信号在隐藏维度上建设性累积，而噪声持续平均抵消。这带来更高的有效信噪比和更平坦的损失曲面，且随着隐藏维度增大，优势愈发显著，表明规模越大越具潜力。参考实现已发布于https://github.com/anonymous452026/ngpt-nvfp4。
原文摘要 · Abstract (English)
Training large language models at 4-bit precision is critical for efficiency. We show that nGPT, an architecture that constrains weights and hidden representations to the unit hypersphere, is inherently more robust to low-precision arithmetic. This removes the need for interventions-such as applying random Hadamard transforms and performing per-tensor scaling calculations-to preserve model quality, and it enables stable end-to-end NVFP4 training. We validate this approach on both a 1.2B dense model and hybrid (Mamba-Transformer) MoE models of up to 3B/30B parameters. We trace this robustness to the dot product: while quantization noise remains largely uncorrelated in both standard and normalized architectures, the signal behaves differently. In nGPT, the hypersphere constraint enhances weak positive correlations among the element-wise products, leading to a constructive accumulation of the signal across the hidden dimension while the noise continues to average out. This yields a higher effective signal-to-noise ratio and a flatter loss landscape, with the effect strengthening as the hidden dimension grows, suggesting increasing advantages at scale. A reference implementation is available at https://github.com/anonymous452026/ngpt-nvfp4
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。