软挑选代softmax，解决注意力坍缩和激活过大问题。
Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
- 采用非归一化修正softmax，避免注意力坍缩
- 340M与1.8B模型均实现0%注意力坍缩率
- 适合低精度量化与稀疏性优化，提升模型可解释性
我们提出softpick，一种不归一化、非求和为1的softmax替代方案，可直接替换Transformer中的注意力机制，有效消除注意力坍缩和巨大激活。在340M和1.8B参数模型上的实验表明，softpick始终实现0%的注意力坍缩率。使用softpick的Transformer生成的隐藏状态具有显著更低的峰度，并产生稀疏的注意力图。量化模型在标准基准测试中表现优于softmax，尤其在低比特精度下优势明显。分析显示，softpick有望为量化、低精度训练、稀疏性优化、剪枝和可解释性带来新可能。
原文摘要 · Abstract (English)
We introduce softpick, a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations. Our experiments with 340M and 1.8B parameter models demonstrate that softpick achieves 0\% sink rate consistently. The softpick transformers produce hidden states with significantly lower kurtosis and creates sparse attention maps. Quantized models using softpick outperform softmax on standard benchmarks, with a particularly pronounced advantage at lower bit precisions. Our analysis and discussion shows how softpick has the potential to open new possibilities for quantization, low-precision training, sparsity optimization, pruning, and interpretability. Our code: https://github.com/zaydzuhri/softpick-attention
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。