改进注意力评分函数，让模型在分布变化下仍能稳定表现
SSA: Improving Performance With a Better Scoring Function
- 用新评分函数SSA替代Softmax，增强上下文学习能力
- 在多个NLP任务上超越使用Softmax的Transformer模型
- 适用于解码器和编码器架构，适合关注泛化性的研究者
尽管Transformer模型具备强大的上下文学习（ICL）能力，但在简单的分布偏移下常出现泛化失败。我们分析了这些失败现象，发现注意力机制中的Softmax评分函数是原因之一。为此，我们提出一种新的注意力评分函数——缩放符号平均（Scaled Signed Averaging, SSA），可有效缓解此类问题。SSA在多项ICL任务中显著提升性能，并在多个NLP基准测试与语言探针任务中优于使用Softmax的Transformer模型，涵盖仅解码器与仅编码器架构。
原文摘要 · Abstract (English)
While transformer models exhibit strong in-context learning (ICL) abilities, they often fail to generalize under simple distribution shifts. We analyze these failures and identify Softmax, the scoring function in the attention mechanism, as a contributing factor. We propose \textbf{Scaled Signed Averaging (SSA)}, a novel attention scoring function that mitigates these failures. SSA significantly improves performance on our ICL tasks and outperforms transformer models with Softmax on several NLP benchmarks and linguistic probing tasks, in both decoder-only and encoder-only architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。