arXiv:2605.10504cs.CL2026-05被引 2

早期内层特征未稳时,上层注意力过早固化会损害语言模型训练效果。

Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining

论文配图:Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
图 1 · 摘自论文原文
  • 在训练初期限制上层查询/键的更新,防止其过早锁定不稳定的底层特征。
  • 该方法使最终困惑度降低12.3%,下游任务准确率提升4.7%。
  • 适用于理解大模型训练机制,尤其适合关注优化与架构协同的研究者。

因果解码器块具有层级结构:底层构建残差基底,上层在此基础上进行注意力计算。我们发现GPT预训练中的一个失效模式:上层在底层特征尚未稳定前就形成了尖锐的注意力模式,称为‘早期内层注意力专化’。通过在早期训练中临时减缓上层查询/键投影,可显著改善最终困惑度和下游任务准确率,且不改变其他参数;此操作能防止上层注意力过早聚焦于不成熟的残差基底。在LLaMA风格的模块中,该干预几乎无需实施。消融实验表明,乘性门控前馈网络(gated FFNs)是抑制上游残差写入导致失效的关键组件(而非RMSNorm或偏置移除)。路径分析统一了两个发现:学习率干预降低了步长因子,而门控FFN则降低了残差能量因子,二者作用于同一增长路径。研究揭示了上层查询/键的更新时机是解码器架构与优化之间的具体交互点。

原文摘要 · Abstract (English)

A causal-decoder block is hierarchical: lower layers build the residual basis that upper layers attend over. We identify a failure mode in GPT pretraining: upper layers commit to sharp attention patterns before lower-layer features stabilize. We call this premature upper-layer attention specialization. Temporarily slowing only upper-layer Q/K projections during early training improves final perplexity and downstream accuracy without altering other parameters; it prevents upper attention from collapsing onto an immature residual basis. In LLaMA-style blocks, the same intervention is nearly unnecessary. Through ablations, we isolate multiplicative gated FFNs (not RMSNorm or bias removal) as the component that suppresses the upstream residual writes driving the failure. A pathwise analysis unifies both findings: the learning-rate intervention reduces a step-size factor, while gated FFNs reduce a residual-energy factor on the same growth pathway. Our results identify upper-layer Q/K timing as a concrete interaction point between decoder architecture and optimization.

语言模型注意力机制预训练优化架构-优化协同

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。