抑制Transformer最后一层隐藏状态突变，提升模型整体性能
Suppressing Final Layer Hidden State Jumps in Transformer Pretraining
- 引入跳跃强度量化指标，捕捉最后一层隐藏状态突变现象
- 提出JREG正则化方法，有效抑制最后一层状态突变
- 在Llama系列模型上验证，无需改架构即提升任务表现
本文研究Transformer语言模型的内部行为，发现许多预训练模型在中间层中输入与输出隐藏状态向量的角距离变化极小，却在最后一层附近出现显著的角距离跃迁。为此，我们首先提出一个定量度量来刻画最后一层附近的跳跃强度，并证明该现象在多个开源模型中普遍存在，且随预训练过程加剧。假设此类跳跃为不良特性，我们提出跳跃抑制正则化（JREG），在预训练阶段对跳跃进行惩罚，以促进中间层能力的均衡使用。基于Llama系列三种规模模型的实证评估显示，采用JREG方法训练的模型在不改变架构的前提下，相比基线模型实现了更优的任务性能。
原文摘要 · Abstract (English)
This paper discusses the internal behavior of Transformer language models. Many recent pre-trained models have been reported to exhibit only slight changes in the angular distance between the input and output hidden state vectors in the middle Transformer layers, despite a disproportionately large ``jump'' in the angular distance occurring in or around the final Transformer layer. To characterize this, we first introduce a quantitative metric for the jump strength around the final layer, and then demonstrate its prevalence across many open-weight models, as well as its amplification throughout pre-training. Assuming such jumps indicate an undesirable property, we propose the jump-suppressing regularizer (JREG) which penalizes this jump during pre-training, thereby encouraging more balanced capability usage across the middle layers. Empirical evaluations of three model sizes of Llama-based models, trained with the proposed JREG method, reveal improved task performance compared to the baseline without altering the model architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。