破解大模型注意力与激活异常，提升量化性能
From Attention to Activation: Unravelling the Enigmas of Large Language Models
- 用改进的softmax-1降低首词注意力过强问题
- 引入OrthoAdam优化器，抑制异常激活值
- 显著提升模型在4比特量化下的稳定性与精度
我们研究了自回归Transformer中的两个奇异现象：（1）注意力头中首词的主导性；（2）隐藏状态中出现的大规模异常激活。发现Llama等主流大模型在98%的注意力头中对首词的注意力达到最大，这归因于softmax函数。为此，我们提出softmax-1重构方案以缓解此问题。同时，识别出自适应优化器（如Adam）是异常激活的主要来源，提出OrthoAdam——通过正交矩阵变换梯度的新优化器来解决该问题。实验表明，这些方法不仅有效抑制上述现象，还能使Transformer在使用基础量化算法时保持性能。具体地，首词注意力比例从65%降至3.3%，隐藏层激活峰度从1657降至3.1，4比特权重量化下的困惑度惩罚从3565降至0.3。
原文摘要 · Abstract (English)
We study two strange phenomena in auto-regressive Transformers: (1) the dominance of the first token in attention heads; (2) the occurrence of large outlier activations in the hidden states. We find that popular large language models, such as Llama attend maximally to the first token in 98% of attention heads, a behaviour we attribute to the softmax function. To mitigate this issue, we propose a reformulation of softmax to softmax-1. Furthermore, we identify adaptive optimisers, e.g. Adam, as the primary contributor to the large outlier activations and introduce OrthoAdam, a novel optimiser that utilises orthogonal matrices to transform gradients, to address this issue. Finally, not only do our methods prevent these phenomena from occurring, but additionally, they enable Transformers to sustain their performance when quantised using basic algorithms, something that standard methods are unable to do. In summary, our methods reduce the attention proportion on the first token from 65% to 3.3%, the activation kurtosis in the hidden states from 1657 to 3.1, and perplexity penalty under 4-bit weight quantisation from 3565 to 0.3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。