将旋转位置编码用于图数据，提升模型对结构信息的捕捉能力。
Rotary Position Encodings for Graphs
- 基于图拉普拉斯谱旋转节点编码，注入结构信息
- 在合成与真实图任务中显著提升性能
- 兼容线性注意力，适合大规模图学习
我们研究了旋转位置编码（RoPE）在图结构数据中的适用性。发现基于图拉普拉斯谱对节点进行旋转编码，能有效将结构信息引入注意力机制，在合成及真实图学习任务中均表现优异。该方法称为_波诱导旋转编码_（WIRE），具有令人关注的理论特性：在网格上可恢复标准RoPE，且渐近依赖于图的有效电阻。与基于偏置的相对位置编码不同，WIRE 兼容线性注意力。
原文摘要 · Abstract (English)
We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data. We find that rotating tokens depending on the spectrum of the graph Laplacian efficiently injects structural information into the attention mechanism, boosting performance in synthetic and real-world graph learning tasks. This approach, coined _Wave-Induced Rotary Encodings_ (WIRE), enjoys intriguing theoretical properties: it recovers regular RoPE on grids, and depends asymptotically on the graph effective resistance. Unlike bias-based relative position encodings, WIRE is compatible with linear attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。