通过过滤无关注意力元素，提升Transformer模型效率与性能。
Selective Attention Improves Transformer
- 引入无参数的有选择性注意力机制，抑制无关上下文影响。
- 相同性能下，参数量可减少至一半，或用更少计算资源达到同等效果。
- 适合追求推理效率、轻量化部署的NLP研究者与工程师。
注意力机制中的冗余信息会降低模型表现。本文提出一种简单且无需参数的改进方法——有选择性注意力，能有效降低对无关内容的关注。该方法在多种模型规模和上下文长度下均显著提升语言建模与下游任务性能。例如，在C4数据集上训练的Transformer使用有选择性注意力时，其语言建模效果相当于标准Transformer使用约两倍注意力头数和参数量的表现。此外，该方法还能大幅减少注意力模块的上下文缓冲区大小，从而显著降低推理阶段的内存与计算开销：在512、1024和2048的上下文长度下，所需内存分别减少16倍、25倍和47倍，且保持相同的验证困惑度。
原文摘要 · Abstract (English)
Unneeded elements in the attention's context degrade performance. We introduce Selective Attention, a simple parameter-free change to the standard attention mechanism which reduces attention to unneeded elements. Selective attention consistently improves language modeling and downstream task performance in a variety of model sizes and context lengths. For example, transformers trained with the language modeling objective on C4 with selective attention perform language modeling equivalently to standard transformers with ~2X more heads and parameters in their attention modules. Selective attention also allows decreasing the size of the attention's context buffer, leading to meaningful reductions in the memory and compute requirements during inference. For example, transformers trained on C4 with context sizes of 512, 1,024, and 2,048 need 16X, 25X, and 47X less memory for their attention module, respectively, when equipped with selective attention, as those without selective attention, with the same validation perplexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。