arXiv:2605.06729cs.LGcs.AI2026-05

提出新型正交残差连接,实现输入自适应且全局保持正交性。

The E$Δ$-MHC-Geo Transformer: Adaptive Geodesic Operations with Guaranteed Orthogonality

论文配图:The E$Δ$-MHC-Geo Transformer: Adaptive Geodesic Operations with Guaranteed Orthogonality
图 1 · 摘自论文原文
  • 融合凯利变换与动态增量学习,构造输入自适应的正交残差模块。
  • 在长序列预测中稳定性达JPmHC的1.9倍、GPT的3.8倍,旋转误差降低4.5倍。
  • 支持精确负反射操作,适合需完整正交群覆盖的高精度建模任务。

本文提出EΔ-MHC-Geo Transformer,通过整合流形约束超连接(mHC)、深度增量学习(DDL)与凯利变换,实现输入自适应且无条件正交的残差连接。不同于仅在β∈{0,2}时正交的Householder算子,本文提出的数据依赖凯利旋转Q(x)=(I+(β/2)A(x))⁻¹(I-(β/2)A(x))对所有β和输入均保持正交。为处理特征值-1的否定情形(凯利变换理论排除),引入EΔ-MHC-Geo混合架构,通过可学习门控机制γ(X)融合凯利旋转与霍尔舍尔反射:X'=γ(X)Q(X)X+(1-γ(X))H₂(X)X。设计中点坍缩正则项4γ(1−γ),激励边界门控决策,确保每种选择组件均正交。在参数量相近(约1.79M/模型)且3次随机种子平均条件下,相比四个基线(含同期的JPmHC),该方法在长时序稳定性上优于JPmHC 1.9倍、优于GPT 3.8倍;单平面近π旋转损失低于JPmHC 4.5倍;范数保持优异,均值偏差仅0.001;诊断反射探针中否定对齐余弦值达0.96,且层数减少33%。尽管JPmHC因更宽表示在纯旋转任务上表现更优,但其有限凯利残差混频器无法实现精确λ=−1算子,且缺乏反射分支,故本文混合设计可访问O(n)的两个连通分支。

原文摘要 · Abstract (English)

We present the E$Δ$-MHC-Geo Transformer, a novel architecture that unifies Manifold-Constrained Hyper-Connections (mHC), Deep Delta Learning (DDL), and the Cayley transform to obtain input-adaptive, unconditionally orthogonal residual connections. Unlike DDL, whose Householder operator is orthogonal only at $β\in \{0,2\}$, our Data-Dependent Cayley rotation $Q(x)=(I+(β/2)A(x))^{-1}(I-(β/2)A(x))$ preserves orthogonality for all $β$ and all inputs. To handle negation, an eigenvalue $-1$ case that Cayley provably excludes, we introduce the E$Δ$-MHC-Geo Hybrid, which combines Cayley rotation with Householder reflection via a learned operator-selection gate $X'=γ(X)Q(X)X+(1-γ(X))H_2(X)X$. A midpoint-collapse regularizer, $4γ(1-γ)$, encourages boundary gate decisions, where each selected component is orthogonal. In matched-parameter comparisons, with approximately 1.79M parameters per model and mean +/- standard deviation over 3 seeds, against four baselines including the concurrent JPmHC, E$Δ$-MHC-Geo achieves the best long-horizon stability, 1.9x over JPmHC and 3.8x over GPT; the best near-$π$ rotation loss, 4.5x over JPmHC on single-plane; strong norm preservation, with 0.001 mean deviation; and 0.96 negation cosine alignment in a diagnostic reflection probe, all with 33% fewer layers. While JPmHC's wider representation excels on pure rotation, its finite Cayley residual mixer excludes an exact $λ=-1$ operator and has no reflection branch, motivating our hybrid approach for accessing both connected components of $O(n)$.

正交网络流形学习深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。