提出非欧注意力框架,实现高效精准的检索与存储。
Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry
- 用非欧几何替代欧氏距离,改进注意力机制的存储与路由
- 理论证明可线性收敛且无局部最优,测试误差受控于η²
- 适合追求理论严谨性的模型设计与高维数据处理场景
我们提出了逆距离注意力的理论基础,从欧氏原型(Resolver)扩展到非欧实现(Riemann GeoResolver)。欧氏部分建立三个核心定理:(1) 电路分离——IDA 实现精确检索仅需 𝒪(1) 资源,而 softmax 需 Ω((log n)²) 宽度;(2) 具有 Ω(e^{Δ²/√d}/Δ²) 更强常数的 Polyak–Lojasiewicz 不等式,暗示线性收敛、在低秩/聚类假设下 𝒪(log n) Lipschitz 缩放、Θ(1) Hessian 散度,且无伪局部极小;(3) 宽度无关的有效秩界,限制噪声记忆——当 d_h ≥ n 时,softmax 可记忆任意标签,而 IDA 将测试误差限于 𝒪(η²)。非欧扩展则以双曲测地距用于存储、球面测地距用于路由。该框架包含十个集成模块:四个覆盖 𝘼(𝑛²) 至 𝘼(1) 每标记的 HIDA 算子;具有可证明误差界的双曲曲率压缩(HCC);具有梯度下界定理的 HyperGate;具有球面类 PL 不等式的球面逆距离注意力(SIDA);具有 𝒪(log T) 损失界动态记忆生成(DMG);以及具有质量与通信边界的测地稀疏路由(GSR)。欧氏定理完整证明,非欧扩展采用类比论证。本工作构建了理论脉络:从欧氏注意力作为特例,到双曲记忆,再到球面检索。
原文摘要 · Abstract (English)
We present a theoretical foundation for inverse-distance attention, from its Euclidean prototype (Resolver) to its non-Euclidean realization (Riemann GeoResolver). The Euclidean part establishes three core theorems: (1) circuit separation---IDA achieves exact retrieval with $\mathcal{O}(1)$ resources while softmax requires $Ω((\log n)^2)$ width; (2) a Polyak--Lojasiewicz inequality with $Ω(e^{Δ^2/\sqrt{d}}/Δ^2)$ stronger constant than softmax, implying linear convergence, $\mathcal{O}(\log n)$ Lipschitz scaling under a low-rank/clustering assumption, $Θ(1)$ Hessian spread, and absence of spurious local minima; (3) a width-independent effective rank bound that limits noise memorization---softmax memorizes arbitrary labels when $d_h\ge n$, while IDA limits test error to $\mathcal{O}(η^2)$. The non-Euclidean extension then builds upon this prototype, replacing Euclidean distance with hyperbolic geodesic distance for storage and spherical geodesic distance for routing. The Riemann GeoResolver framework comprises ten integrated modules: four HIDA operators spanning $Θ(n^2)$ to $Θ(1)$ per token; Hyperbolic Curvature Compression (HCC) with provable error bounds; HyperGate with gradient lower-bound theorem; Spherical Inverse Distance Attention (SIDA) with sphere-analog PL inequalities; Dynamic Memory Genesis (DMG) with $\mathcal{O}(\log T)$ regret bounds; and Geodesic Sparse Routing (GSR) with quality and communication bounds. The Euclidean theorems are proved in full; the non-Euclidean extension theorems are proved with analogous arguments. This work establishes a theoretical arc: from Euclidean attention as a special case, to hyperbolic memory, to spherical retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。