通过特征空间对齐，让稀疏注意力模型性能接近全连接注意力。
SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
- 用双向输出对齐机制融合稀疏与全量注意力
- 在不同稀疏率下均实现顶尖性能，长序列处理能力更强
- 适合需要高效推理的长文本任务
稀疏注意力虽能降低全连接注意力的二次复杂度，但仍面临两大挑战：(1) 注意力差距——在全注意力训练后使用稀疏注意力会导致性能下降，源于训练与推理分布不一致；(2) 能力差距——仅用稀疏注意力训练时梯度流不完整，难以达到全注意力性能。本文提出SSA（Sparse Sparse Attention）训练框架，通过双向注意力输出对齐，联合优化稀疏与全量注意力。理论证明，近似误差随稀疏注意力丢弃的关注质量线性增长，而SSA的对齐目标显著降低了该量值。实验表明，SSA在两种推理模式下均达当前最优性能，可平滑适应不同稀疏预算，并展现出卓越的长上下文处理能力。
原文摘要 · Abstract (English)
Sparse attention reduces the quadratic complexity of full self-attention but faces two challenges: (1) an attention gap, where applying sparse attention to full-attention-trained models causes performance degradation due to train-inference distribution mismatch, and (2) a capability gap, where models trained purely with sparse attention lack complete gradient flow, preventing them from matching full-attention performance. We propose SSA (Sparse Sparse Attention), a training framework that integrates both sparse and full attention with bidirectional attention-output alignment. We prove that the approximation error scales linearly with the attention mass dropped under sparse attention, and show that SSA's alignment objective substantially reduces this quantity compared to baselines. Experiments demonstrate that SSA achieves state-of-the-art performance under both inference modes, adapts smoothly to varying sparsity budgets, and demonstrates superior long-context capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。