提出贝叶斯注意力机制，让模型在超长文本中更准地记住位置信息。
Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
- 将位置编码视为概率模型中的先验，理论统一现有方法
- 在500倍训练长度下仍能准确检索信息，效果优于现有最佳方法
- 仅增加少量参数，适合需要长文本理解的场景
基于Transformer的语言模型依赖位置编码（PE）来处理词序并支持上下文长度外推。然而，现有PE方法缺乏理论清晰性，且评估指标有限，难以验证其外推能力。本文提出贝叶斯注意力机制（BAM），将位置编码形式化为概率模型中的先验。BAM统一了现有方法（如NoPE和ALiBi），并提出了新的广义高斯位置先验，显著提升长上下文泛化能力。实验表明，BAM可在500倍训练上下文长度下实现精准信息检索，在长文本召回准确率上超越当前最优方法，同时保持相近的困惑度，并仅引入极少额外参数。
原文摘要 · Abstract (English)
Transformer-based language models rely on positional encoding (PE) to handle token order and support context length extrapolation. However, existing PE methods lack theoretical clarity and rely on limited evaluation metrics to substantiate their extrapolation claims. We propose the Bayesian Attention Mechanism (BAM), a theoretical framework that formulates positional encoding as a prior within a probabilistic model. BAM unifies existing methods (e.g., NoPE and ALiBi) and motivates a new Generalized Gaussian positional prior that substantially improves long-context generalization. Empirically, BAM enables accurate information retrieval at $500\times$ the training context length, outperforming previous state-of-the-art context length generalization in long context retrieval accuracy while maintaining comparable perplexity and introducing minimal additional parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。