LoKiFormer提升大模型预训练效率，通过显式捕捉局部与全局知识。
LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining
- 用卷积融合注意力，显式建模序列内部局部模式
- 引入可寻址参数记忆库，实现全局知识与计算解耦
- 在预训练中收敛速度比基线快1.33倍，适合高效训练场景
大语言模型在诸多应用中取得突破性进展，但其架构在预训练阶段仍存在效率瓶颈：(i) 自注意力缺乏显式的局部归纳偏置，导致对序列内部局部信息的冗余建模；(ii) 混合专家（MoE）隐式耦合知识存储与计算路径，阻碍对序列外部全局知识的灵活访问。为此，我们提出LoKiFormer，一种新型大模型架构，在标准解码器基础上增加两个专用模块：(1) 局部融合注意力（LFA），通过卷积融合增强注意力机制，显式捕捉局部模式，使注意力作用于更具信息量的表示；(2) 知识记忆模块（KMM），引入可寻址的参数化键值记忆，显式存储全局知识，解耦存储与计算，支持直接知识检索。两者协同实现多层级信息的高效融合。实验表明，LoKiFormer在预训练中收敛速度比基线模型快1.33倍，显著优于现有架构。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved remarkable breakthroughs across various applications. However, their architectures remain inefficient in pretraining due to two main limitations: (i) self-attention lacks an explicit inductive bias for locality, leading to redundant modeling of sequence-internal local information; (ii) mixture-of-experts (MoE) implicitly couples knowledge storage with computational pathways, hindering flexible access to sequence-external global knowledge. To overcome these limitations, we propose LoKiFormer, a novel LLM architecture that augments the standard decoder with two dedicated modules: 1) Local Fusion Attention (LFA), which incorporates a convolutional fusion to attention, explicitly capturing local patterns and allowing the attention to operate on more informative representations; 2) Knowledge Memory Module (KMM), which introduces a parametric key-value memory that explicitly stores global knowledge in addressable slots, decoupling storage from computation and enabling direct knowledge retrieval. Together, these modules enable LoKiFormer to achieve more efficient and effective integration of information at both levels. Experimental results show that LoKiFormer converges 1.33x faster in pre-training than baseline models, underscoring its superiority over existing LLM architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。