arXiv:2607.26775cs.LGcs.AI2026-07

提出多轴结构的旅程算子框架,统一解释位置编码与注意力机制。

Journey Operators for Structured Multi-Axis Composition

  • 用路径上的轴变换乘积定义数据组合与相对位置描述
  • 轴变换可交换时,路径无关性成立,自然导出旋转位置编码
  • 框架可扩展至视觉、语言任务,支持内容自适应的位置偏置

许多数据在一条或多条轴上具有结构:句子中的词、图像中的像素、树中的节点、音频中的帧或3D体数据中的单元。单轴上顺序重要,但独立轴间组合与移动不应依赖轴顺序。本文提出一个建模多轴结构的框架:每个数据项携带其内容及每条轴的微小变换。连接两点的路径定义为旅程;旅程算子是路径上各轴变换的乘积,决定数据沿路径的组合方式及相对位置描述。当变换固定时,该框架恢复旋转位置编码(RoPE)及其多维变体;当变换依赖数据时,模型获得内容自适应的位置归纳偏置。我们证明:仅当各轴变换可交换时,跨轴组合与移动才路径无关。在环面坐标系对称性、柯西条件、双线性及范数保持假设下,推导出配对评分规则必为分块旋转形式,解释了为何类似RoPE的方法自然出现。最后,基于此理论设计了用于值聚合的JoFormer模型,并关联到注意力机制与状态空间模型(SSMs)。初步实验在视觉、语言及长度泛化任务中表明,此类归纳偏置在实践中具有可观测影响。

原文摘要 · Abstract (English)

Many kinds of data have structure along one or more axes: words in a sentence, pixels in an image, nodes in a tree, frames in audio, or cells in a 3D volume. Along one axis, order matters: "the dog bit the man" is different from "the man bit the dog." Across independent axes, however, neither composition nor movement should depend on the order of axes: in an image, composing right then down should give the same result as composing down then right, and moving right then down should describe the same relative position as moving down then right. We develop a framework for modeling this kind of multi-axis structure. Each data item carries its content together with a small transformation for each axis. A path connecting two positions defines a journey; the journey operator is the product of per-axis transformations along that path, governing both how data composes along the path and how relative position is described. When the transformations are fixed, our framework recovers Rotary Position Embedding (RoPE) and its multi-dimensional variants. When they depend on the data, the model gains a content-adaptive positional inductive bias. We show exactly when these paths are well-defined: both composition and movement across axes are path-independent precisely when the axis transformations commute. We also prove that, under the stated toral-frame symmetry, cocycle, bilinearity, and norm-preservation assumptions, the resulting pairwise scoring rule must take the form of block-wise rotations, explaining why RoPE-like methods arise naturally. Finally, we use this theory to design JoFormer, a model for value aggregation, and relate it to attention and state-space models (SSMs). Initial experiments across vision, language, and length generalization suggest that these inductive biases can have observable consequences in practice.

位置编码注意力机制多轴结构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。