揭示Transformer位置编码的数学本质,给出最优编码构造方法。
On the Geometry of Positional Encodings in Transformers
- 从理论上证明位置编码必要性及训练中位置可区分性。
- 提出用希尔伯特距离与MDS构建最优编码,以应力值量化效果。
- 发现最优编码可低秩表示,适合高效实现,适配模型压缩场景。
神经语言模型处理词序列时,内部运算对词序不敏感。位置编码被引入以解决此问题。尽管重要,但现有位置编码多凭试错设计,缺乏数学理论指导。本文建立该理论:第一,无位置信号的Transformer无法解决任何依赖词序的任务(必要性定理);第二,在弱且可验证条件下,所有全局极小值点均使不同位置获得不同向量表示(位置分离定理);第三,通过在位置分布的希尔伯特距离上使用经典多维尺度分析(MDS),构造出信息最优编码的最优近似,编码质量由单一指标‘应力’衡量(命题5,算法1);第四,最优编码有效秩r = rank(B) ≤ n-1,可用r(n+d)参数表示而非nd(最小参数化结果)。附录A在神经切线核(NTK)框架下,对掩码语言建模、序列分类等损失,证明了单调性猜想,基于五个引理。SST-2和IMDB上的实验验证了理论预测,表明线性偏置注意力(ALiBi)的应力显著低于正弦编码与旋转位置编码(RoPE),符合近似平移等变下的秩1解释。
原文摘要 · Abstract (English)
Neural language models process sequences of words, but the mathematical operations inside them are insensitive to the order in which words appear. Positional encodings are the component added to remedy this. Despite their importance, positional encodings have been designed largely by trial and error, without a mathematical theory of what they ought to do. This paper develops such a theory. Four results are established. First, any Transformer without a positional signal cannot solve any task sensitive to word order (Necessity Theorem). Second, training assigns distinct vector representations to distinct sequence positions at every global minimiser, under mild and verifiable conditions (Positional Separation Theorem). Third, the best achievable approximation to an information-optimal encoding is constructed via classical multidimensional scaling (MDS) on the Hellinger distance between positional distributions; the quality of any encoding is measured by a single number, the stress (Proposition 5, Algorithm 1). Fourth, the optimal encoding has effective rank r = rank(B) <= n-1 and can be represented with r(n+d) parameters instead of nd (minimal parametrisation result). Appendix A develops a proof of the Monotonicity Conjecture within the Neural Tangent Kernel (NTK) regime for masked language modelling (MLM) losses, sequence classification losses, and general losses satisfying a positional sufficiency condition, through five lemmas. Experiments on SST-2 and IMDB with BERT-base confirm the theoretical predictions and reveal that Attention with Linear Biases (ALiBi) achieves much lower stress than the sinusoidal encoding and Rotary Position Embedding (RoPE), consistent with a rank-1 interpretation of the MDS encoding under approximate shift-equivariance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。