通过稀疏正则化训练，让注意力矩阵更紧凑高效。
Attention Condensation via Sparsity Induced Regularized Training
- 设计专用损失函数，限制注意力矩阵的非零元素数量。
- GPT-2实验显示，稀疏化后仍能有效捕捉输入依赖关系。
- 适合追求模型推理加速且容忍小幅性能下降的研究者。
随着上下文窗口扩大，自注意力机制在Transformer推理时间中占比越来越高。因此，在最小化性能下降的前提下加速注意力计算，对大语言模型（LLMs）的高效部署至关重要。本文扩展了LLM中注意力稀疏性的理论框架，设计了一种定制化损失函数，通过限制注意力矩阵中前K个最大值的数量来强制稀疏性。在GPT-2上的初步评估表明，采用该损失函数训练的模型，其注意力矩阵既保持稀疏性，又有效捕获相关输入依赖。目前正进一步验证该方法在更大模型和不同架构上的价值。
原文摘要 · Abstract (English)
As the context window expands, self-attention increasingly dominates the transformer's inference time. Therefore, accelerating attention computation while minimizing performance degradation is essential for the efficient deployment of Large Language Models (LLMs). In this study we extend a theoretical framework of attention sparsity in LLMs. A customized loss function is designed to enforce the sparsity by restricting the number of top elements in the attention matrix. We perform an initial set of evaluations with GPT-2 to show the effectiveness of our sparsification approach. The attention matrices of the models trained with the proposed loss are both sparse and effective in capturing relevant input dependencies. We now continue working to demonstrate the value of our approach on larger models and different architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。