提出mHC框架，让超连接网络更稳定可扩展。
mHC: Manifold-Constrained Hyper-Connections
- 将超连接空间投影到特定流形，恢复残差的恒等映射特性
- 训练更稳定，支持大规模模型，性能显著提升
- 适合研究大模型架构与高效神经网络设计者
近期研究如超连接（Hyper-Connections, HC）通过扩展残差流宽度和多样化连接模式，拓展了过去十年广泛使用的残差连接范式。尽管带来显著性能提升，这种多样化从根本上破坏了残差连接固有的恒等映射性质，导致严重训练不稳定和可扩展性受限，并引入显著内存访问开销。为此，我们提出流形约束超连接（Manifold-Constrained Hyper-Connections, mHC），一个通用框架，将HC的残差连接空间投影至特定流形以恢复恒等映射特性，同时结合严格的基础设施优化保障效率。实验表明，mHC能有效支持大规模训练，带来切实性能提升与更优可扩展性。我们预计，mHC作为HC的灵活实用延伸，将促进对拓扑结构设计的深入理解，并为基础模型演进指明有前景的方向。
原文摘要 · Abstract (English)
Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。