提出新型注意力机制，解决长文本建模中注意力衰减与冗余问题。
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
- 通过动态阈值筛选关键词元，实现超稀疏注意力。
- 在长序列下生成超过99%的精确零值，消除注意力黑洞。
- 适合追求高效长文本建模的研究者与工业部署场景。
Softmax注意力因结构限制，在处理长序列时表现不佳：严格的归一化约束导致无关词元产生注意力黑洞，且概率质量随序列增长而分散。本文提出阈值差分注意力（TDA），一种无注意力黑洞的新型机制，可在不增加计算开销的前提下实现超稀疏性并提升长序列鲁棒性。TDA采用行内极值阈值与长度相关门控，仅保留超出阈值的词元；受差分Transformer启发，还引入抑制视图以增强表达能力。理论上证明，TDA将每行伪存活数控制在O(1)量级，且独立视图间共识性伪匹配随上下文增长趋于消失。实验表明，TDA在标准与长序列基准上均实现>99%精确零值，彻底消除注意力黑洞，同时保持竞争力。
原文摘要 · Abstract (English)
Softmax attention struggles with long contexts due to structural limitations: the strict sum-to-one constraint forces attention sinks on irrelevant tokens, and probability mass disperses as sequence lengths increase. We tackle these problems with Threshold Differential Attention (TDA), a sink-free attention mechanism that achieves ultra-sparsity and improved robustness at longer sequence lengths without the computational overhead of projection methods or the performance degradation caused by noise accumulation of standard rectified attention. TDA applies row-wise extreme-value thresholding with a length-dependent gate, retaining only exceedances. Inspired by the differential transformer, TDA also subtracts an inhibitory view to enhance expressivity. Theoretically, we prove that TDA controls the expected number of spurious survivors per row to $O(1)$ and that consensus spurious matches across independent views vanish as context grows. Empirically, TDA produces $>99\%$ exact zeros and eliminates attention sinks while maintaining competitive performance on standard and long-context benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。