用L1距离改进注意力机制，提升语言模型精度与效率
L1 Augmented Attention as an Improved Vector Similarity Metric
- 在点积注意力中加入可学习的L1距离，融合方向与坐标信息
- 在WikiText-2上降低14.5%困惑度，优于RBF L2核方法
- 低维子空间投影减少计算开销，各层头具几何分工特征
缩放点积注意力将方向对齐与向量幅值混杂，限制其作为相似性度量的效果。本文提出L1增强注意力，通过从点积得分中减去查询与键之间可学习的、头独立的L1距离，实现一种混合相似性度量。该方法同时捕捉方向一致性和坐标偏移信息：点积鼓励方向对齐，而L1惩罚坐标差异。为降低L1计算成本，将查询与键投影至低维子空间，其参数专门用于保留有信息量的L1结构。在紧凑Transformer模型上于WikiText-2数据集评估，L1增强注意力相比原始基线实现最高14.5%的困惑度下降，并优于RBF L2核。对范数方差与学习到的L1权重分析显示，不同层间及头级别存在明确的几何角色分化。结果表明，引入L1几何信息能有效且合理地提升现代语言模型中的相似性计算，兼具准确率与并行效率优势。
原文摘要 · Abstract (English)
Scaled dot product attention conflates directional alignment and vector magnitude, limiting its effectiveness as a similarity metric in Transformer models. We introduce L1 augmented attention, a simple and computationally parallelizable modification that subtracts a learned, head specific L1 distance between queries and keys from the dot product score. This hybrid similarity captures complementary geometric information. Dot product rewards directional alignment, while L1 penalizes coordinate deviations. To reduce the cost of L1 computation, we project queries and keys into low dimensional subspaces whose parameters specialize to preserve informative L1 structure. Evaluated on WikiText 2 using a compact transformer, L1 augmented attention achieves up to a 14.5% reduction in perplexity over the original transformer baseline and outperforms an RBF L2 kernel. Analysis of norm variance and learned L1 weights reveals distinct geometric roles across layers and strong head level specialization. These results demonstrate that enriching attention with L1 geometry provides a principled and effective improvement to similarity computation in modern language models, with practical benefits for both accuracy and parallel efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。