提出可学习的相对位置表示空间,提升注意力机制对位置关系的建模能力。
PJ-RoPE: A Fourier-Jet-Affine Position Space for Relative Attention

- 将相对位置建模为可学习的傅里叶-喷流-仿射空间,统一多种位置编码方法。
- 在语言和音乐数据上验证了高阶位置项的有效性,且能稳定训练高阶坐标。
- 适合关注位置建模精度的NLP与序列建模研究者,尤其适用于长序列任务。
我们将注意力中的相对位置机制组织为可学习的傅里叶-喷流-仿射位置空间。出发点是滞后的移位动力学:相对位置核是滞后 $d=i-j$ 的响应函数,一步移位 $(Ef)(d)=f(d+1)$ 通过常系数差分模块对有限结构响应进行紧凑分类。在此视角下,RoPE提供简单的傅里叶根,Jordan-RoPE将这些根扩展为有限傅里叶喷流,ALiBi则提供重复单位根的仿射方向。NTK感知的RoPE缩放具有相同的结构:移动频率网格生成一阶傅里叶喷流切向方向,更高阶泰勒方向生成更高阶喷流。PJ-RoPE显式化并使这些喷流方向可学习,并用所得空间衡量任务级扇区选择。该框架将标量PJ偏置核与精确的PJ旋转特征变换分离,引入扇区门、有效质量、函数能量及留一阶诊断,通过LC/快速度紧致化稳定高阶坐标。受控探测恢复设计的扇区;合成教师显示可训练使用;小字节级语言任务偏好NTK感知的RoPE加仿射近期性;符号音乐令牌流保持LC/仿射变体强健,且可观测高阶修正;LC诊断量化了稳定性与分辨率的权衡。
原文摘要 · Abstract (English)
We organize relative-position mechanisms in attention as a learnable Fourier-Jet-Affine position space. The starting point is lag-shift dynamics: a relative-position kernel is a response function of the lag \(d=i-j\), and the one-step shift \((Ef)(d)=f(d+1)\) gives a compact classification of finite structured responses through constant-coefficient difference modules. In this view, RoPE supplies simple Fourier roots, Jordan-RoPE thickens these roots into finite Fourier jets, and ALiBi supplies the repeated unit-root affine direction. NTK-aware RoPE scaling fits the same structure as a spectral flow of simple Fourier roots: moving the frequency grid generates first Fourier-jet tangent directions, while higher Taylor directions generate higher jets. PJ-RoPE makes these jet directions explicit and learnable, and uses the resulting space to measure task-level sector selection. The framework separates scalar PJ-bias kernels from exact PJ-rotary feature transforms, introduces sector-gate, effective-mass, functional-energy, and leave-one-order-out diagnostics, and stabilizes high-order coordinates with LC/rapidity compactification. Controlled probes recover designed sectors; synthetic teachers show trainable use; small byte-level language runs favor NTK-aware RoPE plus affine recency; symbolic music-token streams keep LC/affine variants strong with measurable high-order corrections; and LC diagnostics quantify the stability-resolution tradeoff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。