通过调节注意力温度，让Transformer更聚焦关键信息，提升长文本处理能力。
Learning to Focus: Focal Attention for Selective and Scalable Transformers
- 用可调温度的softmax sharpen注意力分布，增强重要词权重
- 在长文本任务上相对提升17%至82%，相同精度下参数减少42%
- 适合追求高效推理、长序列建模的NLP应用
注意力是Transformer架构的核心组件，无论编码器、解码器或二者结合模型均依赖它。然而标准softmax注意力常产生噪声分布，影响各层特征选择，尤其在长上下文场景下。本文提出焦点注意力（Focal Attention），通过控制softmax温度来锐化注意力分布，温度可固定或训练时学习。该机制使模型聚焦最相关标记，抑制无关内容。实验表明，相比标准Transformer，Focal Attention在模型规模、训练数据量和上下文长度上均表现更优。跨多个基准测试，其在保持相同准确率前提下，参数量最多减少42%，训练数据量减少33%；在长上下文任务中，相对性能提升达17%至82%，验证了其在实际应用中的有效性。
原文摘要 · Abstract (English)
Attention is a core component of transformer architecture, whether encoder-only, decoder-only, or encoder-decoder model. However, the standard softmax attention often produces noisy probability distribution, which can impair effective feature selection at every layer of these models, particularly for long contexts. We propose Focal Attention, a simple yet effective modification that sharpens the attention distribution by controlling the softmax temperature, either as a fixed hyperparameter or as a learnable parameter during training. This sharpening enables the model to concentrate on the most relevant tokens while suppressing irrelevant ones. Empirically, Focal Attention scales more favorably than standard transformer with respect to model size, training data, and context length. Across diverse benchmarks, it achieves the same accuracy with up to 42% fewer parameters or 33% less training data. On long-context tasks, it delivers substantial relative improvements ranging from 17% to 82%, demonstrating its effectiveness in real world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。