给Transformer加结构化稀疏注意力，反而提升准确率。
Crisp Attention: Regularizing Transformers via Structured Sparsity
- 对DistilBERT的注意力机制施加后验结构稀疏，训练时动态剪枝。
- 80%稀疏度下准确率达91.59%，比稠密模型高0.97个百分点。
- 稀疏性像正则化器，防止过拟合，适合追求高效高精度模型的场景。
自注意力机制的二次计算开销是扩展Transformer模型的主要挑战。尽管注意力稀疏性被广泛研究以提升计算效率，但普遍认为会牺牲模型准确率。本文报告了一个反直觉的现象：在SST-2情感分析任务上，对DistilBERT模型的注意力机制施加结构化、事后稀疏化后，模型准确率显著提升。80%注意力稀疏度下，验证准确率达到91.59%，较稠密基线绝对提升0.97%。我们推测该现象源于稀疏性作为强隐式正则化器，通过限制特征使用范围，迫使模型基于更受约束且稳健的特征进行预测。本工作将注意力稀疏性从单纯提升效率的工具，重新定义为改善Transformer泛化能力与性能的潜在手段。
原文摘要 · Abstract (English)
The quadratic computational cost of the self-attention mechanism is a primary challenge in scaling Transformer models. While attention sparsity is widely studied as a technique to improve computational efficiency, it is almost universally assumed to come at the cost of model accuracy. In this paper, we report a surprising counter-example to this common wisdom. By introducing structured, post-hoc sparsity to the attention mechanism of a DistilBERT model during fine-tuning on the SST-2 sentiment analysis task, we find that model accuracy improves significantly. Our model with 80\% attention sparsity achieves a validation accuracy of 91.59\%, a 0.97\% absolute improvement over the dense baseline. We hypothesize that this phenomenon is due to sparsity acting as a powerful implicit regularizer, preventing the model from overfitting by forcing it to make predictions with a more constrained and robust set of features. Our work recasts attention sparsity not just as a tool for computational efficiency, but as a potential method for improving the generalization and performance of Transformer models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。