arXiv:2601.05732cs.LGcs.AI2026-01被引 12

mHC-lite用置换矩阵组合构造双随机矩阵,无需迭代就能稳定训练。

mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations

  • 用置换矩阵的凸组合显式构造双随机矩阵,保证数学精确性。
  • 在多个数据集上性能持平或优于mHC,训练吞吐量更高。
  • 仅需普通矩阵运算,无需复杂CUDA代码,易部署且无稳定性问题。

超连接(HC)通过动态残差矩阵在多条残差流间混合信息,加速深层网络收敛。然而,未受约束的残差矩阵可能影响训练稳定性。为解决此问题,DeepSeek提出流形约束超连接(mHC),通过迭代Sinkhorn-Knopp(SK)归一化将矩阵近似投影到Birkhoff多面体。我们发现该方法存在两方面局限:(i) 有限次SK迭代无法保证精确双随机性,误差随网络深度累积,削弱稳定性;(ii) 高效的SK实现依赖高度定制化的CUDA核函数,增加工程门槛并降低可移植性。基于Birkhoff-von Neumann定理,我们提出mHC-lite,一种简单的重参数化方法,显式将双随机矩阵构建为置换矩阵的凸组合。该方法从构造上保证精确双随机性,仅需原生矩阵操作即可实现。大量实验表明,mHC-lite在性能上达到或超过mHC,同时以简单实现获得更高训练吞吐量,并消除了HC与mHC中观察到的残差不稳定性。代码已公开于https://github.com/FFTYYY/mhc-lite。

原文摘要 · Abstract (English)

Hyper-Connections (HC) generalizes residual connections by introducing dynamic residual matrices that mix information across multiple residual streams, accelerating convergence in deep neural networks. However, unconstrained residual matrices can compromise training stability. To address this, DeepSeek's Manifold-Constrained Hyper-Connections (mHC) approximately projects these matrices onto the Birkhoff polytope via iterative Sinkhorn--Knopp (SK) normalization. We identify two limitations of this approach: (i) finite SK iterations do not guarantee exact doubly stochasticity, leaving an approximation gap that can accumulate through network depth and undermine stability; (ii) efficient SK implementation requires highly specialized CUDA kernels, raising engineering barriers and reducing portability. Motivated by the Birkhoff--von Neumann theorem, we propose mHC-lite, a simple reparameterization that explicitly constructs doubly stochastic matrices as convex combinations of permutation matrices. This approach guarantees exact doubly stochasticity by construction and can be implemented using only native matrix operations. Extensive experiments demonstrate that mHC-lite matches or exceeds mHC in performance while achieving higher training throughput with a naive implementation and eliminating the residual instabilities observed in both HC and mHC. The code is publicly available at https://github.com/FFTYYY/mhc-lite.

神经网络残差连接双随机矩阵模型稳定

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。