提出新方法实现高表达力的约束混合，提升网络稳定性与扩展性。
TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes
- 基于运输多面体构造精确双随机混合矩阵，自由度为(n-1)²
- 在语言模型预训练中性能媲美现有方法，训练更稳定且可扩展
- 避免迭代归一化和组合爆炸，适合大规模深度学习应用
超连接（HC）通过在多个残差流间引入可学习混合来提升残差网络性能，但无约束混合会导致训练不稳定。流形约束超连接（mHC）通过Sinkhorn归一化近似满足双随机性，而mHC-lite通过置换矩阵的凸组合确保精确约束，但计算复杂度为阶乘级。KromHC使用克罗内克积参数化降低复杂度，但将混合矩阵限制在Birkhoff多面体的结构子流形上。本文提出运输Birkhoff多面体（TBP）参数化及其递归变体（RTBP），构造具有精确双随机性的混合矩阵，自由度为(n-1)²。该方法避免迭代归一化和组合爆炸，同时保持Birkhoff多面体的完整表达能力。在语言模型预训练中的实验证明，该方法具有竞争力的性能，且训练更稳定、可扩展性更强。
原文摘要 · Abstract (English)
Hyper-Connections (HC) improve residual networks by introducing learnable mixing across multiple residual streams, but unconstrained mixing leads to training instability. Manifold-Constrained Hyper-Connections (mHC) address this by enforcing approximate double stochasticity via Sinkhorn normalization, while mHC-lite ensures exact constraints through convex combinations of permutation matrices at the cost of factorial complexity. KromHC reduces this cost using Kronecker-product parameterizations, but restricts the mixing matrices to a structured submanifold of the Birkhoff polytope . We propose Transportation Birkhoff Polytope (TBP) parameterizations and their Recursive variants (RTBP), which construct exactly doubly stochastic mixing matrices with $(n-1)^2$ degrees of freedom. Our approach avoids iterative normalization and combinatorial explosion while preserving full expressivity of the Birkhoff polytope. Empirical results on language model pre-training' demonstrate competitive performance with improved stability and scalability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。