让大模型像人一样按需读上下文，提升效率不降性能。
Learning When Not to Attend Globally
- 每注意力头用二元路由动态切换全局与局部注意力
- 256词窗口下93%全局注意力可替换为局部窗口无性能损失
- 发现上下文依赖呈长尾分布，局部处理已足够多数场景
阅读书籍时，人类主要关注当前页面，仅在必要时回溯前文。类似地，我们证明大型语言模型可学会动态决定何时访问全局上下文。提出全或此处注意力（AHA），每个注意力头使用二元路由器，动态切换全注意力与局部滑动窗口注意力。结果表明，在256词窗口下，最多93%的原始全注意力操作可被滑动窗口注意力替代而性能不变。通过在不同窗口大小下评估AHA，我们发现上下文依赖呈长尾分布，随着局部窗口扩大，全注意力需求迅速衰减。解耦局部处理与全局访问后，AHA揭示全注意力大多冗余，高效推理只需按需获取全局上下文。
原文摘要 · Abstract (English)
When reading books, humans focus primarily on the current page, flipping back to recap prior context only when necessary. Similarly, we demonstrate that Large Language Models (LLMs) can learn to dynamically determine when to attend to global context. We propose All-or-Here Attention (AHA), which utilizes a binary router per attention head to dynamically toggle between full attention and local sliding window attention for each token. Our results indicate that with a window size of 256 tokens, up to 93\% of the original full attention operations can be replaced by sliding window attention without performance loss. Furthermore, by evaluating AHA across various window sizes, we identify a long-tail distribution in context dependency, where the necessity for full attention decays rapidly as the local window expands. By decoupling local processing from global access, AHA reveals that full attention is largely redundant, and that efficient inference requires only on-demand access to the global context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。