arXiv:2603.20896cs.LGcs.AI2026-03被引 3

提出新型残差连接,提升模型表达力且训练更稳定。

Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections

  • 用谱球约束替代传统多面体约束,允许负值参数。
  • 解决身份退化问题,实现更强的跨流特征交互。
  • 避免不稳定迭代与高复杂度参数化,适合深层网络设计。

Hyper-Connections(HC)将残差连接推广至多流结构,通过残差矩阵实现跨流特征混合以增强模型表达力。然而,无约束混合会破坏残差连接固有的身份映射特性,导致训练不稳定。为此,已有方法如mHC及其变体通过Sinkhorn迭代或基于排列的参数化将残差矩阵限制在双随机矩阵构成的Birkhoff多面体中。我们揭示该约束存在三大缺陷:(1) 身份退化,即学习矩阵坍缩至单位矩阵,削弱跨流交互;(2) 表达瓶颈,因非负性限制无法实现减法式特征解耦;(3) 参数化低效,表现为不稳定的小型迭代或排列参数化带来的阶乘级开销。为克服上述局限,本文提出谱球约束的Hyper-Connections(sHC)。通过将可行集从刚性的多面体几何转向谱范数球,使残差矩阵可含负值,从而支持选择性特征分化所需的减法交互。该设计消除了不稳定的Sinkhorn投影和阶乘级参数化,实现了表达丰富、非退化的残差矩阵,同时保持训练稳定性。

原文摘要 · Abstract (English)

Hyper-Connections (HC) generalize residual connections into multiple streams, employing residual matrices for cross-stream feature mixing to enrich model expressivity. However, unconstrained mixing disrupts the identity mapping property intrinsic to the residual connection, causing unstable training. To address this, Manifold-Constrained Hyper-Connections (mHC) and its variant restrict these matrices to the Birkhoff polytope (doubly stochastic matrices) via Sinkhorn iterations or permutation-based parameterizations. We reveal three limitations of this polytope constraint: (1) identity degeneration, where learned matrices collapse around the identity and diminish cross-stream interactions, (2) an expressivity bottleneck, as the non-negativity constraint prevents subtractive feature disentanglement, and (3) parameterization inefficiencies, manifesting as unstable Sinkhorn iterations or the factorial-scaling overhead of permutation-based parameterizations. To overcome these flaws, we propose Spectral-Sphere-Constrained Hyper-Connections (sHC). By geometrically shifting the feasible set from a rigid polytope to a spectral norm sphere, sHC allows negative entries, unlocking subtractive interactions for selective feature diversification. This shift eliminates unstable Sinkhorn projections and factorial parameterization, enabling expressive, non-degenerate residual matrices while preserving training stability.

神经网络残差连接谱约束特征融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。