arXiv:2605.04217cs.LGcs.CL2026-05被引 1

提出新型相对位置编码Jordan-RoPE,融合振荡与多项式特征提升注意力建模。

Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks

  • 用复数若尔当块构造非半单位置编码,生成振荡-多项式特征。
  • 在小规模WikiText-103任务中,改进版优于RoPE和直接求和基线。
  • 适合建模距离调制相位交互的场景,结构优势显著但不普遍超越。

相对位置编码决定了查询-键滞后函数如何进入原始注意力得分。RoPE提供旋转变相,ALiBi则添加距离偏置。受线性平移不变位置编码的群论视角启发,本文研究了一种非半单情形:复数旋转变相本征值与幂零响应共存于同一缺陷若尔当块。由此产生的相对算子生成振荡-多项式特征,如 $e^{-γd} ext{cos}(ωd)$、$e^{-γd} ext{sin}(ωd)$、$d e^{-γd} ext{cos}(ωd)$、$d e^{-γd} ext{sin}(ωd)$($d=i-j/geq 0$ 为因果滞后)。该构造实现距离调制的相位基 $d e^{iωd}$,而非简单叠加距离通道。我们提出精确若尔当-RoPE作为一参数非半单表示,给出其实数块形式,并明确非正交位置映射所需的对偶查询作用。还区分了精确表示与数值更稳定的稳定变体——后者虽改善数值行为,但破坏精确群律。核级诊断与若尔当友好型合成语言任务表明,耦合若尔当基在目标含距离调制相位交互时有效。在小型WikiText-103字节语言模型上,缩放版精确变体优于同族基准,但整体仍逊于RoPE+ALiBi。

原文摘要 · Abstract (English)

Relative positional encodings determine which functions of query-key lag can enter the primitive attention logit. RoPE supplies a rotary phase, while ALiBi supplies an additive distance bias. Motivated by group-theoretic views of linear translation-invariant positional encodings, we study a non-semisimple case in which a complex rotary eigenvalue and a nilpotent response live in the same defective Jordan block. The resulting relative operator generates oscillatory-polynomial features such as $e^{-γd}\cos(ωd)$, $e^{-γd}\sin(ωd)$, $d e^{-γd}\cos(ωd)$, and $d e^{-γd}\sin(ωd)$, for causal lag $d=i-j\geq 0$. Thus the construction realizes a distance-modulated phase basis $d e^{iωd}$, rather than merely adding a separate distance channel to RoPE. We formulate Exact Jordan-RoPE as a non-semisimple one-parameter representation, give its real block form, and specify the contragredient query action required by non-orthogonal positional maps. We also distinguish this exact representation from stabilized variants whose bounded shear improves numerical behavior but breaks the exact group law. Kernel-level diagnostics and a Jordan-friendly synthetic language-model task show that the coupled Jordan basis is useful when the target contains distance-modulated phase interactions. On a small WikiText-103 byte language model, a scaled-exact variant improves over RoPE and direct-sum baselines within the Jordan family, while RoPE+ALiBi remains strongest overall. The evidence is structural rather than a broad performance claim.

位置编码注意力机制若尔当块

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。