通过预评分优先计算重要关键词，提升长文本建模精度。
Efficient Attention via Pre-Scoring: Prioritizing Informative Keys in Transformers
- 用聚类或杠杆评分法预先标记关键信息，只计算重要键值对。
- 在13.1万词长上下文上，困惑度从12.0降至9.5，保持高效。
- 适用于语言与视觉模型，适合追求高精度的长序列任务。
高效注意力机制虽能支持长上下文变换器，但常忽略全局重要标记，降低建模质量。本文提出预评分框架，在应用分层近似注意力前，为键值分配与查询无关的全局重要性先验。通过基于聚类或杠杆式的评分方法，识别出结构上有信息量的键，并将计算限制在该优先子集内。与HyperAttention结合后，在长上下文语言建模中显著提升近似质量：在131,000词上下文的ChatGLM上，固定交互预算下困惑度从12.0降至9.5，同时保持亚二次效率。在相同键数预算下，聚类评分始终优于杠杆选择。该方法在视觉变换器中替换自注意力也基本保留基准准确率，表明其跨模态泛化能力。我们在植入选项模型下提供结构保证，证明聚类可恢复与杠杆方法相同的高权重键集合。总体而言，预评分通过更优的关键信息优先策略，在不牺牲可扩展性的前提下，改善了近似注意力的效率-精度权衡。
原文摘要 · Abstract (English)
Efficient attention mechanisms enable long-context transformers but often miss globally important tokens, degrading modeling quality. We introduce a pre-scoring framework that assigns a query-independent global importance prior to keys before applying hierarchical approximate attention. Using clustering-based or leverage-style scoring, pre-scoring identifies structurally informative keys and restricts computation to this prioritized subset. Integrated with HyperAttention, pre-scoring substantially improves approximation quality on long-context language modeling: on ChatGLM with 131k-token contexts, perplexity decreases from 12.0 to 9.5 under a fixed interaction budget while retaining subquadratic efficiency. Clustering-based scoring consistently outperforms leverage-based selection under identical key budgets. Beyond language, replacing self-attention in Vision Transformers preserves most of the baseline accuracy, showing that the approach generalizes across modalities. We provide structural guarantees under a planted-subspace model, showing that clustering recovers the same heavy-key sets as leverage-based methods. Overall, pre-scoring improves the efficiency-accuracy trade-off of approximate attention by better prioritizing informative keys without sacrificing scalability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。