通过中心化输出嵌入，解决大模型预训练中的日志发散问题。
Output Embedding Centering for Stable LLM Pretraining
- 从输出嵌入几何角度分析，发现各向异性是不稳定根源。
- 提出OEC方法，可有效抑制日志发散，且在有无权重共享下均稳定。
- 相比z-loss更少依赖超参数调优，适合实际部署场景。
大语言模型的预训练不仅成本高昂，还容易出现训练不稳定性。尤其在训练末期常出现输出日志发散问题。当前主流缓解策略如z-loss和日志软截断仅治标不治本。本文从输出嵌入的几何特性出发，识别出各向异性嵌入是根本原因。基于此，提出输出嵌入中心化（OEC）新策略，证明其能有效抑制日志发散。OEC可通过两种方式实现：一种是确定性操作μ-中心化，另一种是正则化方法μ-loss。实验表明，两种变体在训练稳定性上均优于z-loss，与日志软截断相当，且在存在或不存在权重共享时均表现一致。作为次要发现，μ-loss对正则化超参数的敏感度显著低于z-loss。
原文摘要 · Abstract (English)
Pretraining of large language models is not only expensive but also prone to certain training instabilities. A specific instability that often occurs at the end of training is output logit divergence. The most widely used mitigation strategies, z-loss and logit soft-capping, merely address the symptoms rather than the underlying cause of the problem. In this paper, we analyze the instability from the perspective of the output embeddings' geometry and identify anisotropic embeddings as its source. Based on this, we propose output embedding centering (OEC) as a new mitigation strategy, and demonstrate that it suppresses output logit divergence. OEC can be implemented in two different ways: as a deterministic operation called $μ$-centering, or a regularization method called $μ$-loss. Our experiments show that both variants outperform z-loss in terms of training stability, while being on par with logit soft-capping. This holds true both in the presence and the absence of weight tying. As a secondary result, we find that $μ$-loss is significantly less sensitive to regularization hyperparameter tuning than z-loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。