提出KV移位注意力，让语言模型更高效学习归纳能力。
KV Shifting Attention Enhances Language Modeling
- 用KV移位注意力替代传统多层注意力，降低对深度和宽度的要求。
- 在从玩具模型到100亿参数以上预训练模型中均实现更快收敛或更好性能。
- 适合研究高效注意力机制或提升大模型上下文学习能力的读者。
当前大型语言模型主要基于仅解码结构的Transformer，具备出色的上下文学习（ICL）能力。普遍认为其ICL能力的基础是归纳头机制，该机制通常需要至少两层注意力。为更高效地实现模型的归纳能力，本文重新审视归纳头机制，提出一种KV移位注意力。理论证明，该方法可降低模型对归纳头机制在深度与宽度上的要求。实验结果表明，KV移位注意力有助于学习归纳头与语言建模，在从玩具模型到超过100亿参数的预训练模型上均表现出更好的性能或更快的收敛速度。
原文摘要 · Abstract (English)
The current large language models are mainly based on decode-only structure transformers, which have great in-context learning (ICL) capabilities. It is generally believed that the important foundation of its ICL capability is the induction heads mechanism, which requires at least two layers attention. In order to more efficiently implement the ability of the model's induction, we revisit the induction heads mechanism and proposed a KV shifting attention. We theoretically prove that the KV shifting attention reducing the model's requirements for the depth and width of the induction heads mechanism. Our experimental results demonstrate that KV shifting attention is beneficial to learning induction heads and language modeling, which lead to better performance or faster convergence from toy models to the pre-training models with more than 10 B parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。