提出新型位置编码HoPE，解决长文本建模中注意力衰减不稳问题。
HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models
- 用双曲几何中的洛伦兹旋转重构位置编码，取代传统旋转编码
- 在扩展序列长度测试中，困惑度显著低于现有方法，稳定提升长程依赖建模能力
- 适合需要处理超长文本的场景，如法律文书、学术论文生成
位置编码使Transformer能够建模文本的序列结构和长距离依赖。绝对位置编码因固定表示难以外推到更长序列，相对编码如Alibi在极长上下文中性能下降，而广泛使用的旋转位置编码（RoPE）引入振荡注意力模式，阻碍稳定的长距离依赖建模。本文通过几何重释位置编码，受双曲几何中洛伦兹变换启发，提出双曲旋转位置编码（HoPE），利用双曲函数实现对标记表示的洛伦兹旋转。理论分析表明，RoPE是本方法的特例。HoPE从根本上解决了RoPE的不稳定问题，强制注意力权重随标记距离单调衰减。大量实验结果，包括多个扩展序列基准上的困惑度评估，表明HoPE持续优于现有位置编码方法。这些发现凸显了其在表示和泛化长距离依赖方面的增强能力。数据与代码将公开。
原文摘要 · Abstract (English)
Positional encoding mechanisms enable Transformers to model sequential structure and long-range dependencies in text. While absolute positional encodings struggle with extrapolation to longer sequences due to fixed positional representations, and relative approaches like Alibi exhibit performance degradation on extremely long contexts, the widely-used Rotary Positional Encoding (RoPE) introduces oscillatory attention patterns that hinder stable long-distance dependency modelling. We address these limitations through a geometric reformulation of positional encoding. Drawing inspiration from Lorentz transformations in hyperbolic geometry, we propose Hyperbolic Rotary Positional Encoding (HoPE), which leverages hyperbolic functions to implement Lorentz rotations on token representations. Theoretical analysis demonstrates that RoPE is a special case of our generalized formulation. HoPE fundamentally resolves RoPE's slation issues by enforcing monotonic decay of attention weights with increasing token distances. Extensive experimental results, including perplexity evaluations under several extended sequence benchmarks, show that HoPE consistently exceeds existing positional encoding methods. These findings underscore HoPE's enhanced capacity for representing and generalizing long-range dependencies. Data and code will be available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。