通过正交超连接实现梯度稳定，提升深层网络训练效率。
JPmHC Dynamical Isometry via Orthogonal Hyper-Connections
- 用可学习线性混合器替代恒等映射，控制梯度条件。
- 在ARC-AGI上收敛更快、精度更高、计算成本更低。
- 适合追求稳定高效训练的深度模型研究者。
近期深度学习进展，如超连接（Hyper-Connections, HC），通过引入更宽的残差流和多样化的连接模式扩展了残差连接范式。然而，这些创新牺牲了残差连接的恒等映射特性，导致训练不稳定、可扩展性受限及内存开销增加。为此，本文提出JPmHC（Jacobian谱保持的流形约束超连接），将恒等跳跃替换为作用于n条并行流的可学习线性混合器，并显式控制梯度条件。通过将混合器M约束在算子范数有界的流形上（如双随机、Stiefel、Grassmann），JPmHC避免梯度病态，增强稳定性。主要贡献包括：(i) 基于自由概率的分析，预测结构化跳跃下的Jacobian谱，提供混合器选择的设计准则；(ii) 采用内存高效的隐式微分进行固定点投影，降低激活内存与同步开销；(iii) 通过Cayley变换实现Stiefel约束混合器，保证正交性无需事后归一化。在ARC-AGI上的实证表明，相比双随机基线，JPmHC实现更快收敛、更高精度和更低计算成本，秩-p的Grassmann变体表现介于两者之间——与谱理论预测一致。作为一种灵活可扩展的HC延伸，JPmHC推动了谱感知、稳定且高效的深度学习发展，为拓扑架构设计与基础模型演进提供洞见。
原文摘要 · Abstract (English)
Recent advances in deep learning, exemplified by Hyper-Connections (HC), have expanded the residual connection paradigm by introducing wider residual streams and diverse connectivity patterns. While these innovations yield significant performance gains, they compromise the identity mapping property of residual connections, leading to training instability, limited scalability, and increased memory overhead. To address these challenges, we propose JPmHC (Jacobian-spectrum Preserving manifold-constrained Hyper-Connections), a framework that replaces identity skips with a trainable linear mixer acting on n parallel streams while explicitly controlling gradient conditioning. By constraining the mixer M on operator-norm-bounded manifolds (e.g., bistochastic, Stiefel, Grassmann), JPmHC prevents gradient pathologies and enhances stability. JPmHC introduces three key contributions: (i) a free-probability analysis that predicts Jacobian spectra for structured skips, providing actionable design rules for mixer selection; (ii) memory-efficient implicit differentiation for fixed-point projections, reducing activation memory and synchronization overhead; and (iii) a Stiefel-constrained mixer via Cayley transforms, ensuring orthogonality without post-hoc normalization. Empirical evaluations on ARC-AGI demonstrate that JPmHC achieves faster convergence, higher accuracy, and lower computational cost compared to bistochastic baselines, with a rank-$p$ Grassmannian variant tracking between the two -- consistent with the spectral theory predictions. As a flexible and scalable extension of HC, JPmHC advances spectrum-aware, stable, and efficient deep learning, offering insights into topological architecture design and foundational model evolution. \newline \newline
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。