用筛选机制让注意力模型更精准，减少参数还更快
Screening Is Enough
- 引入筛选机制，直接判断键值相关性并剔除无关项
- 参数少30%且训练学习率更高，长文本表现更稳
- 适合追求高效长文本处理的场景，尤其看重推理速度
标准softmax注意力无法提供独立可解释的查询-键相关性度量：注意力分数无界，权重仅相对于竞争键定义。因此，无关键无法被明确拒绝，即使无真正相关键也分配注意力。我们提出Multiscreen，一种基于筛选机制的语言模型架构，实现绝对查询-键相关性。筛选不重新分配所有键的注意力，而是计算有界的查询-键相似度，并应用显式阈值，剔除无关键，对剩余键聚合而不进行全局竞争。实验显示，Multiscreen在验证损失上与Transformer基线相当，参数量减少约30%，可在更大学习率下保持稳定。其在远超训练上下文长度的长文本中仍保持稳定困惑度，检索性能随上下文增长几乎无下降。此外，在长上下文场景下，全上下文前向传播延迟更低。
原文摘要 · Abstract (English)
A core limitation of standard softmax attention is that it does not provide an independently interpretable measure of query--key relevance: attention scores are unbounded, while attention weights are defined only relative to competing keys. Consequently, irrelevant keys cannot be explicitly rejected, and some attention mass is assigned even when no key is genuinely relevant. We introduce Multiscreen, a language-model architecture built around a mechanism we call screening, which enables absolute query--key relevance. Instead of redistributing attention across all keys, screening computes bounded query--key similarities and applies an explicit threshold, discarding irrelevant keys and aggregating the remaining keys without global competition. Across experiments, Multiscreen achieves comparable validation loss with roughly 30\% fewer parameters than a Transformer baseline and remains stable at substantially larger learning rates. It maintains stable long-context perplexity beyond the training context and shows little degradation in retrieval performance as context length increases. Finally, Multiscreen achieves lower full-context forward-pass latency at long context lengths.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。