通过阈值筛选让Transformer推理更高效，不损失精度。
Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding
- 用固定阈值自动保留每行关键注意力元素，无需重训练。
- 推理时显存占用降低3-10倍，注意力计算量减少最多10倍。
- 适用于对效率要求高的NLP部署，尤其适合资源受限场景。
我们提出Top-Theta（Top-$θ$）注意力，一种无需训练的Transformer推理阶段稀疏化方法。核心思想是为每个注意力头设置静态阈值，使每行保留固定数量的关键元素。该方法实现基于内容的稀疏性，且在不同数据域下保持鲁棒性。我们进一步引入补偿机制，在极端稀疏化下仍能保持精度，确立阈值剪枝作为top-k注意力的实用且合理替代方案。在自然语言处理任务上进行了广泛评估，结果显示，Top-$θ$可实现3-10倍的V-cache使用减少，推理时注意力元素数量减少最多10倍，精度下降不超过1%。
原文摘要 · Abstract (English)
We present Top-Theta (Top-$θ$) Attention, a training-free method for sparsifying transformer attention during inference. Our key insight is that static, per-head thresholds can be calibrated to retain the desired constant number of significant elements per attention row. This approach enables content-based sparsity without retraining, and it remains robust across data domains. We further introduce compensation techniques to preserve accuracy under aggressive sparsification, establishing attention thresholding as a practical and principled alternative to top-k attention. We provide extensive evaluation on natural language processing tasks, showing that Top-$θ$ achieves 3-10x reduction in V-cache usage and up to 10x fewer attention elements during inference while degrading no more than 1% in accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。